Camera configuration on movable objects

ABSTRACT

Systems and methods for obstacle detection and state information determination are provided. In some embodiments, a movable object may carry one or more imaging devices. The imaging devices may be arranged on the movable object so as to have a field of view oriented vertically relative to the movable object. The arrangement of the imaging device may complement or supplant existing arrangement schemes and provide efficient, multi-functional and cost-effective means of arranging imaging devices on movable objects.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application a continuation of International Application No.PCT/CN2015/083517, filed Jul. 8, 2015, the disclosure of which is herebyincorporated by reference in its entirety.

BACKGROUND

Movable objects such as unmanned aerial vehicles (UAVs) can be used forperforming surveillance, reconnaissance, and exploration tasks in a widevariety of environments for military and civilian applications. A UAVmay be manually controlled by a remote user, or may operate in asemi-autonomous or fully autonomous manner. Such UAVs can includesensors configured to collect data from the surrounding environment andprocessors to process the data for use in various applications (e.g.,navigation).

Sensors may be arranged on the movable objects in a variety ofconfigurations. Existing approaches for the arrangement of sensors onmovable objects may be less than optimal in some instances. For example,information that is acquired by the sensors may be limited by theirplacement relative to the movable object. The utility of a sensor may belimited by the arrangement of the sensor on the movable objects.

SUMMARY

Embodiments disclosed herein provide systems and methods for obstacledetection and state information determination. In many embodiments,imaging devices may be arranged on a movable object so as to have afield of view oriented vertically relative to the movable object. Imagedata obtained using the imaging devices may be processed by one or moreprocessors for further use. Advantageously, the approaches describedherein may provide improved obstacle detection, obstacle avoidance,navigation, and determination of state information for movable objects.

Thus in one aspect, a method of detecting obstacles using a movableobject is provided. The method comprises: receiving image data of anobstacle from an imaging device carried by the movable object, whereinthe imaging device is configured to have a field of view oriented upwardor downward relative to the movable object; receiving attitudeinformation for the movable object; and calculating, via one or moreprocessors, a distance between the movable object and the obstacle basedon the image data of the obstacle and the attitude information for themovable object.

In some embodiments, the movable object comprises a plurality ofpropulsion units configured to effect movement of the movable object. Insome embodiments, the movable object is an unmanned aerial vehicle(UAV). In some embodiments, the plurality of propulsion units comprisesa plurality of rotors. In some embodiments, the imaging device is acamera. In some embodiments, the camera is a monocular camera. In someembodiments, the imaging device comprises an optical axis substantiallyparallel to a vertical axis of the movable object. In some embodiments,the field of view of the imaging device is about 60 degrees. In someembodiments, the field of view of the imaging device is oriented upwardrelative to the movable object. In some embodiments, the method furthercomprises operating the movable object within an indoor environment,wherein the image data comprises images of a ceiling within the indoorenvironment. In some embodiments, the calculating step comprisesdifferentiating between a portion of the image data depicting theceiling and a portion of the image data depicting the obstacle. In someembodiments, within the indoor environment, the distance from themovable object to the obstacle is estimated based on an angle of view ofthe obstacle within the images. In some embodiments, the method furthercomprises operating the movable object within an outdoor environment,wherein the image data comprises images of the sky. In some embodiments,the calculating step comprises differentiating between a portion of theimage data depicting the sky and a portion of the image data depictingthe obstacle. In some embodiments, within the outdoor environment, thedistance from the movable object to the obstacle is estimated based onan angle of view of the obstacle within the images. In some embodiments,the field of view of the imaging device is oriented downward relative tothe movable object. In some embodiments, the movable object is operatedwithin an indoor environment and the image data comprises images of afloor within the indoor environment. In some embodiments, the image datacomprises images of the floor meeting a wall. In some embodiments, thedistance from the movable object to the obstacle is estimated based onan angle of view of the obstacle within the images, wherein a negativeangle value is applied in the estimation. In some embodiments, theattitude information is generated by an inertial measurement unit (IMU).In some embodiments, the attitude information comprises at least one ofa roll angle, a pitch angle, or a yaw angle of the movable object. Insome embodiments, the calculating step comprises determining a portionof the field of view of the imaging device that is occupied by theobstacle, based on the image data.

In another aspect, a system for detecting obstacles using movable objectis provided. The system comprises: an imaging device carried by themovable object and configured to obtain image data of an obstacle,wherein the imaging device is configured to have a field of vieworiented upward or downward relative to the movable object; and one ormore processors individually or collectively configured to: receive theimage data of the obstacle from the imaging device; receive attitudeinformation for the movable object; and calculate a distance between themovable object and the obstacle based on the image data of the obstacleand the attitude information for the movable object.

In some embodiments, the movable object comprises a plurality ofpropulsion units configured to effect movement of the movable object. Insome embodiments, the movable object is an unmanned aerial vehicle(UAV). In some embodiments, the plurality of propulsion units comprisesa plurality of rotors. In some embodiments, the imaging device is acamera. In some embodiments, the camera is a monocular camera. In someembodiments, the imaging device comprises an optical axis substantiallyparallel to a vertical axis of the movable object. In some embodiments,the field of view of the imaging device is about 60 degrees. In someembodiments, the field of view of the imaging device is oriented upwardrelative to the movable object. In some embodiments, the movable objectis operated within an indoor environment and the image data comprisesimages of a ceiling within the indoor environment. In some embodiments,the one or more processors are configured to differentiate between aportion of the image data depicting the ceiling and a portion of theimage data depicting the obstacle. In some embodiments, within theindoor environment, the distance from the movable object to the obstacleis estimated based on an angle of view of the obstacle within theimages. In some embodiments, the movable object is operated within anoutdoor environment and the image data comprises images of the sky. Insome embodiments, the one or more processors are individually orcollectively configured to differentiate between a portion of the imagedata depicting the sky and a portion of the image data depicting theobstacle. In some embodiments, within the outdoor environment, thedistance from the movable object to the obstacle is estimated based onan angle of view of the obstacle within the images. In some embodiments,the field of view of the imaging device is oriented downward relative tothe movable object. In some embodiments, the movable object is operatedwithin an indoor environment and the image data comprises images of afloor within the indoor environment. In some embodiments, the image datacomprises images of the floor meeting a wall. In some embodiments, thedistance from the movable object to the obstacle is estimated based onan angle of view of the obstacle within the images, wherein a negativeangle value is applied in the estimation. In some embodiments, theattitude information is generated by an inertial measurement unit (IMU).In some embodiments, the attitude information comprises at least one ofa roll angle, a pitch angle, or a yaw angle of the movable object. Insome embodiments, the one or more processors are individually orcollectively configured to determine a portion of the field of view ofthe imaging device that is occupied by the obstacle, based on the imagedata.

In another aspect, a method of detecting obstacles using a movableobject is provided. The method comprises: receiving image data from animaging device carried by the movable object, wherein the imaging deviceis configured to have a field of view oriented upward or downwardrelative to the movable object, and wherein the image data comprisesimage data of an obstacle positioned at a side of the movable object;and calculating, via one or more processors, a distance between themovable object and the obstacle based on the image data of the obstacle.

In some embodiments, the movable object comprises a plurality ofpropulsion units configured to effect movement of the movable object. Insome embodiments, the movable object is an unmanned aerial vehicle(UAV). In some embodiments, the plurality of propulsion units comprisesa plurality of rotors. In some embodiments, the imaging device is acamera. In some embodiments, the camera is a monocular camera. In someembodiments, the imaging device comprises an optical axis substantiallyparallel to a vertical axis of the movable object. In some embodiments,the field of view of the imaging device is about 60 degrees. In someembodiments, the field of view of the imaging device is oriented upwardrelative to the movable object. In some embodiments, the method furthercomprises operating the movable object within an indoor environment,wherein the image data comprises images of a ceiling within the indoorenvironment. In some embodiments, the calculating step comprisesdifferentiating between a portion of the image data depicting theceiling and a portion of the image data depicting the obstacle. In someembodiments, within the indoor environment, the distance from themovable object to the obstacle is estimated based on an angle of view ofthe obstacle within the images. In some embodiments, the method furthercomprises operating the movable object within an outdoor environment,wherein the image data comprises images of the sky. In some embodiments,the calculating step comprises differentiating between a portion of theimage data depicting the sky and a portion of the image data depictingthe obstacle. In some embodiments, within the outdoor environment, thedistance from the movable object to the obstacle is estimated based onan angle of view of the obstacle within the images. In some embodiments,the field of view of the imaging device is oriented downward relative tothe movable object. In some embodiments, the method further comprisesoperating the movable object within an indoor environment, wherein theimage data comprises images of a floor within the indoor environment. Insome embodiments, the image data comprises images of the floor meeting awall. In some embodiments, the distance from the movable object to theobstacle is estimated based on an angle of view of the obstacle withinthe images, wherein a negative angle value is applied in the estimation.In some embodiments, the obstacle is located in front of the movableobject, behind the movable object, to the left of the movable object, tothe right of the movable object, or combinations thereof. In someembodiments, the calculating step comprises determining a portion of thefield of view of the imaging device that is occupied by the obstacle,based on the image data.

In another aspect, a system for detecting obstacles using a movableobject is provided. The system comprises: an imaging device carried bythe movable object and configured to obtain image data, wherein theimaging device is configured to have a field of view oriented upward ordownward relative to the movable object, and wherein the image datacomprises image data of an obstacle positioned at a side of the movableobject; and one or more processors individually or collectivelyconfigured to: receive the image data from the imaging device; andcalculate a distance between the movable object and the obstacle basedon the image data of the obstacle.

In some embodiments, the movable object comprises a plurality ofpropulsion units configured to effect movement of the movable object. Insome embodiments, the movable object is an unmanned aerial vehicle(UAV). In some embodiments, the plurality of propulsion units comprisesa plurality of rotors. In some embodiments, the imaging device is acamera. In some embodiments, the camera is a monocular camera. In someembodiments, the imaging device comprises an optical axis substantiallyparallel to a vertical axis of the movable object. In some embodiments,the field of view of the imaging device is about 60 degrees. In someembodiments, the field of view of the imaging device is oriented upwardrelative to the movable object. In some embodiments, the movable objectis operated within an indoor environment and the image data comprisesimages of a ceiling within the indoor environment. In some embodiments,the one or more processors are individually or collectively configuredto differentiate between a portion of the image data depicting theceiling and a portion of the image data depicting the obstacle. In someembodiments, within the indoor environment, the distance from themovable object to the obstacle is estimated based on an angle of view ofthe obstacle within the images. In some embodiments, the movable objectis operated within an outdoor environment and the image data comprisesimages of the sky. In some embodiments, the one or more processors areindividually or collectively configured to differentiate between aportion of the image data depicting the sky and a portion of the imagedata depicting the obstacle. In some embodiments, within the outdoorenvironment, the distance from the movable object to the obstacle isestimated based on an angle of view of the obstacle within the images.In some embodiments, the field of view of the imaging device is orienteddownward relative to the movable object. In some embodiments, themovable object is operated within an indoor environment and the imagedata comprises images of a floor within the indoor environment. In someembodiments, the image data comprises images of the floor meeting awall. In some embodiments, the distance from the movable object to theobstacle is estimated based on an angle of view of the obstacle withinthe images, wherein a negative angle value is applied in the estimation.In some embodiments, the obstacle is located in front of the movableobject, behind the movable object, to the left of the movable object, tothe right of the movable object, or combinations thereof. In someembodiments, the one or more processors are individually or collectivelyconfigured to determine a portion of the field of view of the imagingdevice that is occupied by the obstacle, based on the image data.

In another aspect, a method of detecting obstacles using an imagingdevice carried by a movable object is provided. The method comprises:receiving, at one or more processors, first image data of an obstaclefrom the imaging device carried by the movable object, wherein theimaging device is arranged on the movable object so as to have a fieldof view oriented upward or downward relative to the movable object;generating, at the one or more processors, a control signal forcontrolling one or more propulsion units of the movable object to causethe movable object to move; receiving, at the one or more processors,second image data of the obstacle from the imaging device after themovable object has moved; and calculating, with aid of the one or moreprocessors, a distance between the movable object and the obstacle basedon the first image data and the second image data.

In some embodiments, the movable object is an unmanned aerial vehicle(UAV). In some embodiments, the one or more propulsion units comprises aplurality of rotors. In some embodiments, the imaging device is acamera. In some embodiments, the camera is a monocular camera. In someembodiments, the imaging device comprises an optical axis substantiallyparallel to a vertical axis of the movable object. In some embodiments,the field of view of the imaging device is about 60 degrees. In someembodiments, the field of view of the imaging device is oriented upwardrelative to the movable object. In some embodiments, the method furthercomprises operating the movable object within an indoor environment,wherein the image data comprises images of a ceiling within the indoorenvironment. In some embodiments, the field of view of the imagingdevice is oriented downward relative to the movable object. In someembodiments, the method further comprises operating the movable objectwithin an indoor environment, wherein the image data comprises images ofa floor within the indoor environment. In some embodiments, the imagedata comprises images of the floor meeting a wall. In some embodiments,the control signals cause the movable object to move a horizontaldistance. In some embodiments, the horizontal distance is about 0.2 m to1 m. In some embodiments, the control signals cause the movable objectto move a vertical distance. In some embodiments, the vertical distanceis about 0.2 m to 1 m. In some embodiments, the calculating stepcomprises assessing the first and second image data in order to detectchanges in a portion of the field of view of the imaging device occupiedby the obstacle prior to and after the movable object has moved. In someembodiments, the method further comprises determining a horizontaldistance or a vertical distance moved. In some embodiments, thehorizontal or vertical distance moved is determined using an inertialmeasurement unit (IMU).

In another aspect, a system for detecting obstacles using a movableobject is provided. The system comprises: an imaging device carried bythe movable object and configured to obtain image data of an obstacle,wherein the imaging device is arranged on the movable object so as tohave a field of view oriented upward or downward relative to the movableobject; one or more propulsion units configured to effect movement ofthe movable object; and one or more processors individually orcollectively configured to: receive first image data of the obstaclefrom the imaging device; generate control signals for controlling theone or more propulsion units to cause the movable object to move;receive second image data of the obstacle from the imaging device afterthe movable object has moved; and calculate a distance between themovable object and the obstacle based on the first image data and thesecond image data.

In some embodiments, the one or more processors are individually orcollectively configured to determine a height of the obstacle above themovable object, based on the first image data and the second image data.In some embodiments, the distance between the movable object and theobstacle is determined based on the height of the obstacle above themovable object. In some embodiments, the movable object is an unmannedaerial vehicle (UAV). In some embodiments, the one or more propulsionunits comprises a plurality of rotors. In some embodiments, the imagingdevice is a camera. In some embodiments, the camera is a monocularcamera. In some embodiments, the imaging device comprises an opticalaxis substantially parallel to a vertical axis of the movable object. Insome embodiments, the field of view of the imaging device is about 60degrees. In some embodiments, the field of view of the imaging device isoriented upward relative to the movable object. In some embodiments, themovable object is operated within an indoor environment and the imagedata comprises images of a ceiling within the indoor environment. Insome embodiments, the field of view of the imaging device is orienteddownward relative to the movable object. In some embodiments, themovable object is operated within an indoor environment and the imagedata comprises images of a floor within the indoor environment. In someembodiments, the image data comprises images of the floor meeting awall. In some embodiments, the control signals cause the movable objectto move a horizontal distance. In some embodiments, the horizontaldistance is about 0.2 m to 1 m. In some embodiments, the control signalscause the movable object to move a vertical distance. In someembodiments, the vertical distance is about 0.2 m to 1 m. In someembodiments, the one or more processors are individually or collectivelyconfigured to assess the first and second image data in order to detectchanges in a portion of the field of view of the imaging device occupiedby the obstacle prior to and after the movable object has moved. In someembodiments, the one or more processors are individually or collectivelyconfigured to determine a horizontal distance or a vertical distancemoved. In some embodiments, the horizontal or vertical distance moved isdetermined using an inertial measurement unit (IMU).

In another aspect, a method for determining state information for amovable object is provided. The method comprises: receiving, at one ormore processors, image data from an imaging device carried by themovable object, wherein the imaging device is arranged on the movableobject so as to have a field of view oriented upwards relative to themovable object; receiving, at the one or more processors, attitudeinformation for the movable object; and calculating, with aid of the oneor more processors, at least one of position information or motioninformation for the movable object based on the image data and theattitude information for the movable object.

In some embodiments, the movable object comprises a plurality ofpropulsion units configured to effect movement of the movable object. Insome embodiments, the movable object is an unmanned aerial vehicle(UAV). In some embodiments, the plurality of propulsion units comprisesa plurality of rotors. In some embodiments, the imaging device is acamera. In some embodiments, the camera is a monocular camera. In someembodiments, the imaging device comprises an optical axis substantiallyparallel to a vertical axis of the movable object. In some embodiments,the field of view of the imaging device is about 60 degrees. In someembodiments, the method further comprises operating the movable objectwithin an indoor environment, wherein the image data comprises images ofa ceiling within the indoor environment. In some embodiments, theattitude information is generated by an inertial measurement unit (IMU).In some embodiments, the attitude information comprises at least one ofa roll angle, a pitch angle, or a yaw angle of the movable object. Insome embodiments, the motion information comprises velocity of themovable object. In some embodiments, the image data comprises a sequenceof images. In some embodiments, the calculating step comprises:detecting feature points in each image of the sequence of images; anddetermining changes in the feature points over the sequence of images.In some embodiments, the method further comprises generating anenvironmental map using the movable object. In some embodiments, theenvironmental map is generated using Simultaneous Localization andMapping (SLAM) techniques. In some embodiments, the position informationis calculated relative to the generated map. In some embodiments, themethod further comprises outputting control signals to control positionor motion of the movable object, based on the calculated at least one ofposition information or motion information.

In another aspect, a system for determining state information for amovable object is provided. The system comprises: an imaging devicecarried by the movable object and configured to obtain image data,wherein the imaging device is arranged on the movable object so as tohave a field of view oriented upwards relative to the movable object;and one or more processors individually or collectively configured to:receive the image data from the imaging device; receive attitudeinformation for the movable object; and calculate at least one ofposition information or motion information for the movable object basedon the image data and the attitude information for the movable object.

In some embodiments, the movable object comprises a plurality ofpropulsion units configured to effect movement of the movable object. Insome embodiments, the movable object is an unmanned aerial vehicle(UAV). In some embodiments, the plurality of propulsion units comprisesa plurality of rotors. In some embodiments, the imaging device is acamera. In some embodiments, the camera is a monocular camera. In someembodiments, the imaging device comprises an optical axis substantiallyparallel to a vertical axis of the movable object. In some embodiments,the field of view of the imaging device is about 60 degrees. In someembodiments, the movable object is operated within an indoor environmentand the image data comprises images of a ceiling within the indoorenvironment. In some embodiments, the attitude information is generatedby an inertial measurement unit (IMU). In some embodiments, the attitudeinformation comprises at least one of a roll angle, a pitch angle, or ayaw angle of the movable object. In some embodiments, the motioninformation comprises velocity of the movable object. In someembodiments, the image data comprises a sequence of images. In someembodiments, the one or more processors are individually or collectivelyconfigured to: detect feature points in each image of the sequence ofimages; and determine changes in the feature points over the sequence ofimages. In some embodiments, the one or more processors are individuallyor collectively configured to generate an environmental map using themovable object. In some embodiments, the environmental map is generatedusing Simultaneous Localization and Mapping (SLAM) techniques. In someembodiments, the position information is calculated relative to thegenerated map. In some embodiments, the one or more processors areindividually or collectively configured to output control signals tocontrol position or motion of the movable object, based on thecalculated at least one of position information or motion information.

It shall be understood that different aspects of the invention can beappreciated individually, collectively, or in combination with eachother. Various aspects of the invention described herein may be appliedto any of the particular applications set forth below. Other objects andfeatures of the present invention will become apparent by a review ofthe specification, claims, and appended figures.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWING

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1 illustrates a UAV with an imaging device oriented upwards, inaccordance with embodiments.

FIG. 2 illustrates a schematic of obstacle detection and avoidance, inaccordance with embodiments.

FIG. 3 illustrates a movable object with an imaging device moving avertical distance, in accordance with embodiments.

FIG. 4 illustrates a movable object with an imaging device moving ahorizontal distance, in accordance with embodiments.

FIG. 5 illustrates a method of detecting obstacles using a movableobject, in accordance with embodiments.

FIG. 6 illustrates a method of detecting obstacles using an imagingdevice carried by a movable object, in accordance with embodiments.

FIG. 7 illustrates a method for determining state information for amovable object, in accordance with embodiments.

FIG. 8 illustrates a method of detecting obstacles using a movableobject, in accordance with embodiments.

FIG. 9 illustrates a UAV operating in an outdoor environment inaccordance with embodiments.

FIG. 10 illustrates a UAV operating in an indoor environment inaccordance with embodiments.

FIG. 11 illustrates an unmanned aerial vehicle (UAV), in accordance withembodiments.

FIG. 12 illustrates a movable object including a carrier and a payload,in accordance with embodiments.

FIG. 13 is a schematic illustration by way of block diagram of a systemfor controlling a movable object, in accordance with embodiments.

DETAILED DESCRIPTION

The present disclosure provides systems and methods for obstacledetection and state information determination for movable objects. Amovable object, such as an unmanned aerial vehicle (UAV) may be capableof processing information about its environment with the aid of sensors.Arrangement of sensors on a UAV may determine the functionality of thesensors. For example, sensors such as cameras on a UAV may be used tocapture image data of the environment. Cameras on the UAV orienteddownwards may be utilized in determining state information of the UAV.Cameras on the UAV oriented laterally or sideways may be used to detectobstacles. Cameras on the UAV oriented upwards may be utilized indetermining state information of the UAV and/or in detecting andavoiding obstacles. Arrangements provided herein may enable a singlesensor to be used for a plurality of functions. For example, avertically oriented camera may be used to both detect obstacles anddetermine state information of the UAV.

In some embodiments, a UAV can be adapted to carry one or more sensors.The one or more sensors may be configured to collect relevant data, suchas information relating to the UAV's state, the surrounding environment,or the objects and obstacles within the environment. The relevant datamay be analyzed, processed, or be used in further applications. Forexample, based on the relevant data that is collected, it can bepossible to generate control signals for controlling UAV navigation.Exemplary sensors suitable for use with the embodiments disclosed hereininclude location sensors (e.g., global positioning system (GPS) sensors,mobile device transmitters enabling location triangulation), visionsensors (e.g., imaging devices capable of detecting visible, infrared,or ultraviolet light, such as cameras), proximity or range sensors(e.g., ultrasonic sensors, lidar, time-of-flight or depth cameras),inertial sensors (e.g., accelerometers, gyroscopes, inertial measurementunits (IMUs)), altitude sensors, attitude sensors (e.g., compasses)pressure sensors (e.g., barometers), audio sensors (e.g., microphones)or field sensors (e.g., magnetometers, electromagnetic sensors). Anysuitable number and combination of sensors can be used, such as one,two, three, four, five, six, seven, eight, or more sensors. Optionally,the data can be received from sensors of different types (e.g., two,three, four, five, six, seven, eight, or more types). Sensors ofdifferent types may measure different types of signals or information(e.g., position, orientation, velocity, acceleration, proximity,pressure, etc.) and/or utilize different types of measurement techniquesto obtain data. For instance, the sensors may include any suitablecombination of active sensors (e.g., sensors that generate and measureenergy from their own energy source) and passive sensors (e.g., sensorsthat detect available energy). As another example, some sensors maygenerate absolute measurement data that is provided in terms of a globalcoordinate system (e.g., position data provided by a GPS sensor,attitude data provided by a compass or magnetometer), while othersensors may generate relative measurement data that is provided in termsof a local coordinate system (e.g., relative angular velocity providedby a gyroscope; relative translational acceleration provided by anaccelerometer, relative attitude information provided by a visionsensor, relative distance information provided by an ultrasonic sensor,lidar, or time-of-flight camera). In some instances, the localcoordinate system may be a body coordinate system that is definedrelative to the UAV.

The sensors can be configured to collect various types of data, such asdata relating to the UAV, the surrounding environment, or objects withinthe environment. For example, at least some of the sensors may beconfigured to provide data regarding a state of the UAV. The stateinformation provided by a sensor can include information regarding aspatial disposition of the UAV (e.g., location or position informationsuch as longitude, latitude, and/or altitude; orientation or attitudeinformation such as roll, pitch, and/or yaw). The state information canalso include information regarding motion of the UAV (e.g.,translational velocity, translational acceleration, angular velocity,angular acceleration, etc.). A sensor can be configured, for instance,to determine a spatial disposition and/or motion of the UAV with respectto up to six degrees of freedom (e.g., three degrees of freedom inposition and/or translation, three degrees of freedom in orientationand/or rotation). The state information may be provided relative to aglobal coordinate system or relative to a local coordinate system. Aglobal coordinate system may refer to a coordinate system independent toa location of the UAV or another entity. A local coordinate system mayrefer to a coordinate system relative to the UAV or another entity. Forexample, a sensor can be configured to determine the distance betweenthe UAV and the user controlling the UAV, or the distance between theUAV and the starting point of flight for the UAV.

The data obtained by the sensors may provide various types ofenvironmental information. For example, the sensor data may beindicative of an environment type, such as an indoor environment,outdoor environment, low altitude environment, or high altitudeenvironment. The sensor data may also provide information regardingcurrent environmental conditions, including weather (e.g., clear, rainy,snowing), visibility conditions, wind speed, time of day, and so on.Furthermore, the environmental information collected by the sensors mayinclude information regarding the objects in the environment, such asthe obstacles described herein or landmarks that are recognizable by aprocessor. Obstacle information may include information regarding thenumber, density, geometry, spatial disposition, movement, trajectory,and/or velocity of obstacles in the environment.

In some embodiments, a UAV can include one or more vision sensors, alsoreferred to herein as “imaging devices.” While many embodiments aredescribed herein as having one imaging device coupled to a UAV, it is tobe understood that any number of imaging devices may be coupled to aUAV, such as one, two, three, four, five or more imaging devices. Animaging device can be configured to detect electromagnetic radiation(e.g., visible, infrared, and/or ultraviolet light) and generate imagedata based on the detected electromagnetic radiation. For example, animaging device may include a charge-coupled device (CCD) sensor or acomplementary metal-oxide-semiconductor (CMOS) sensor that generateselectrical signals in response to wavelengths of light. The resultantelectrical signals can be processed to produce image data. The imagedata generated by an imaging device can include one or more images,which may be static images (e.g., photographs), dynamic images (e.g.,video), or suitable combinations thereof. The image data can bepolychromatic (e.g., RGB, CMYK, HSV) or monochromatic (e.g., grayscale,black-and-white, sepia).

A sequence of two or more images may be captured by an imaging device.For example, a sequence of about 2, 3, 4, 5, 10, 15, 20, 25, 50, 100,150, 200, or more images may be captured by the imaging device. Theimaging device may capture a sequence of images at a specific capturerate. In some embodiments, the sequence of images may be captured atstandard video frame rates such as about 24p, 25p, 30p, 48p, 50p, 60p,72p, 90p, 100p, 120p, 300p, 50i, or 60i. In some embodiments, thesequence of images may be captured at a rate less than or equal to aboutone image every 0.0001 seconds, 0.0002 seconds, 0.0005 seconds, 0.001seconds, 0.002 seconds, 0.005 seconds, 0.002 seconds, 0.05 seconds, 0.01seconds, 0.02 seconds, 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds. In someembodiments, the capture rate may change depending on user input and/orexternal conditions (e.g. rain, snow, wind, texture of environment beingcaptured).

In some embodiments, the imaging device can be a camera. A camera can bea movie or video camera that captures dynamic image data (e.g., video).A camera can be a still camera that captures static images (e.g.,photographs). A camera can be a binocular camera. A binocular camera asused herein may refer to a stereo, or a stereovision camera. A stereocamera may comprise two cameras. A camera may be a monocular camera.Although certain embodiments provided herein are described in thecontext of cameras, it shall be understood that the present disclosurecan be applied to any suitable imaging device, and any descriptionherein relating to cameras can also be applied to any suitable imagingdevice, and any description herein relating to cameras can also beapplied to other types of imaging devices. A camera can be used togenerate 2D images of a 3D scene (e.g., an environment, one or moreobjects, etc.). The images generated by the camera can represent theprojection of the 3D scene onto a 2D image plane. Accordingly, eachpoint in the 2D image corresponds to a 3D spatial coordinate in thescene.

Images obtained by the imaging devices described herein can be used fora variety of applications related to UAV operation. In some embodiments,the images are used to facilitate UAV navigation within an environment(e.g., autonomously, semi-autonomously, or manually). In someembodiments, the images are used for obstacle detection and avoidance.In some embodiments, the images can be processed to assess or determinestate information for the UAV (e.g., position, orientation, velocity,and/or acceleration information). The state information can be assessedor determined using (e.g., processing with one or more processors) oneor more images obtained by one or more imaging devices.

The imaging devices may each have a field of view. The field of view ofan imaging device may be the extent of the environment that isdetectable (e.g., visible) by the imaging device. The field of view maybe related to the angle of view, which may be measured by the angularextent of a given scene that is imaged by the imaging device. The angleof view of an imaging device may be at an angle of less than or about360°, 300°, 240°, 180°, 150°, 120°, 90°, 60°, 30°, 20°, or 10°. Thefield of view may be described by the relative direction of the imagingdevice to the movable object. For example, the field of view may beoriented vertically, horizontally, upward, downward, side-ways, and thelike relative to the movable object (e.g., a UAV). The imaging devicesmay each have an optical axis. The optical axis of an imaging device,which may also be referred to as the “principal axis,” can be a linealong which there is some degree of rotational symmetry in the imagingdevice. In some embodiments, the optical axis of the imaging devicepasses through the center of the optical components (e.g., lens, photosensor) of the imaging device.

Imaging devices of the present disclosure can be situated on anysuitable portion of a movable object, such as above, underneath, on theside(s) of, or within a body of the movable object. Some imaging devicescan be mechanically coupled to the UAV such that the spatial dispositionand/or motion of the movable object correspond to the spatialdisposition and/or motion of the imaging device. The imaging devices canbe coupled to the movable object via a rigid coupling, such that theimaging device does not move relative to the portion of the movableobject to which it is attached. Alternatively, the coupling between theimaging device and the movable object can permit movement (e.g.,translational or rotational movement relative to the UAV) of the imagingdevice relative to the movable object. The movement of the imagingdevice relative to the movable object may be controlled remotely by userinput, autonomously, or semi-autonomously. The coupling can be apermanent coupling or nonpermanent (e.g., releasable) coupling. Suitablecoupling methods can include adhesives, bonding, welding, and/orfasteners (e.g., screws, nails, pins, etc.). Optionally, the imagingdevice can be integrally formed with a portion of the movable object.Furthermore, the imaging device can be electrically coupled with aportion of the movable object (e.g., processing unit, control system,data storage) so as to enable the data collected by the imaging deviceto be used for various functions of the UAV (e.g., navigation, control,propulsion, communication with a user or other device, etc.), such asthe embodiments discussed herein. The imaging device may be operablycoupled with a portion of the UAV (e.g., processing unit, controlsystem, data storage).

The arrangement and/or orientation of the imaging devices relative tothe movable object (e.g., a UAV) may determine the utility andlimitations of imaging devices. In some embodiments, imaging devicessituated underneath the UAV (e.g., an imaging device with a field ofview oriented downward relative to the UAV) may be used to determine aspeed of the UAV. For example, a sequence of one or more images may becaptured by the imaging device, and a change in the relative position ofcoordinates on the floor may be analyzed within the captured images todetermine a speed of the UAV. In some embodiments, imaging devicessituated on the side of the UAV (e.g., an imaging device with a field ofview oriented frontward, sideward, backward, or laterally relative tothe UAV) may be used for obstacle detection and/or avoidance. Aprocessor may determine an obstacle by determining depth information byevaluating information from a sensor. In some methods, for obstacledetection or obstacle avoidance, two or more sensors may be situated onthe side of the UAV. For example, an ultrasonic sensor may be used inconcert with an imaging device, or two imaging devices may be usedtogether for obstacle detection and/or avoidance. In some embodiments,the imaging device may measure the relative movement of the UAVaccording to an optical flow of two consecutive images, and recover theactual movement (e.g., distance moved by the UAV) based on measurementof an ultrasonic sensor. In some embodiments, two imaging devices or abinocular camera (e.g., stereovision camera) may be used in concert witha processor to determine a distance to an object. For a UAV havingcapabilities of obstacle detection and/or avoidance and speedmeasurement, three or more sensors may be used (e.g., an imaging devicesituated on the side of the UAV, an imaging device situated underneaththe UAV, and an ultrasonic sensor).

FIG. 1 illustrates a UAV 100 with an imaging device 102 orientedupwards, in accordance with embodiments. The imaging device 102 may bearranged on the movable object so as to have a field of view orientedvertically relative to the movable object. “Vertically oriented” may beused herein to refer to an optical axis of the imaging device beingaligned with and/or substantially parallel to a vertical axis 112 of themovable object. “Vertical axis” may be used herein to refer to the axisextending from the upper surface (e.g., sky-facing surface) to the lowersurface (e.g., ground-facing surface) of the movable object. Thus, thevertical axis of the movable object may vary with an attitude of themovable object. For example, if the movable object tilts 90 degreesrelative to a horizontal floor, the vertical axis of the movable objectmay also tilt 90 degrees relative to the horizontal floor. In this case,for an imaging device fixed relative to the movable object (e.g.,imaging device 102 relative to UAV 100), the imaging device may nolonger have a field of view oriented vertically relative to theenvironment (e.g., horizontal floor). However, the imaging device maynonetheless have a field of view oriented vertically relative to themovable object. As used herein, an imaging device arranged on themovable object so as to have a field of view oriented verticallyrelative to the movable object may refer to an imaging device arrangedon the movable object so as to have a field of view oriented upward ordownward relative to the movable object and the two terms may be usedinterchangeably throughout.

An imaging device arranged on the movable object so as to have a fieldof view oriented upward or downward (e.g., field of view orientedvertically) relative to the movable object may have an optical axisexactly parallel to a vertical axis of the movable object. An imagingdevice arranged on the movable object so as to have a field of vieworiented upward or downward (e.g., field of view oriented vertically)relative to the movable object may have an optical axis substantiallyparallel to a vertical axis of the movable object. An optical axissubstantially parallel to a vertical axis of the movable object mayrefer to an optical axis within 5°, 10°, 15°, 20°, 25°, 30°, 35°, 40°,or 45° of the vertical axis of the movable object. An imaging devicearranged on the movable object so as to have a field of view orientedupward or downward relative to the movable object may have an opticalaxis within 0 degrees, 5 degrees, 10 degrees, 20 degrees, 30 degrees, 40degrees, 60 degrees, or 80 degrees of the vertical axis of the movableobject. In some instances, an imaging device oriented on the movableobject so as to have a field of view oriented upward relative to themovable object may be oriented on or above an upper surface (e.g.,sky-facing surface) of the movable object. The imaging device orientedon the movable object so as to have a field of view oriented upwardrelative to the movable object may have an optical axis exactly parallelto a vertical axis of the movable object. The imaging device oriented onthe movable object so as to have a field of view oriented upwardrelative to the movable object may have an optical axis substantiallyparallel to a vertical axis of the movable object. An optical axissubstantially parallel to a vertical axis of the movable object mayrefer to an optical axis within 5°, 10°, 15°, 20°, 25°, 30°, 35°, 40°,or 45° of the vertical axis of the movable object. The imaging deviceoriented on the movable object so as to have a field of view orientedupward relative to the movable object may have an optical axis within 0degrees, 5 degrees, 10 degrees, 20 degrees, 30 degrees, 40 degrees, 60degrees, or 80 degrees of the vertical axis of the movable object. Insome instances, an imaging device oriented on the movable object so asto have a field of view oriented downward relative to the movable objectmay be oriented on or below a lower surface (e.g., ground-facingsurface) of the movable object. The imaging device oriented on themovable object so as to have a field of view oriented downward relativeto the movable object may have an optical axis exactly parallel to avertical axis of the movable object. The imaging device oriented on themovable object so as to have a field of view oriented downward relativeto the movable object may have an optical axis substantially parallel toa vertical axis of the movable object. An optical axis substantiallyparallel to a vertical axis of the movable object may refer to anoptical axis within 5°, 10°, 15°, 20°, 25°, 30°, 35°, 40°, or 45° of thevertical axis of the movable object. The imaging device oriented on themovable object so as to have a field of view oriented downward relativeto the movable object may have an optical axis within 0 degrees, 5degrees, 10 degrees, 20 degrees, 30 degrees, 40 degrees, 60 degrees, or80 degrees of the vertical axis of the movable object.

An imaging device oriented on the movable object so as to have a fieldof view oriented upward or downward relative to the movable object mayhave an optical axis above a lateral axis 114 (or longitudinal axis) ofthe movable object or below a lateral (or longitudinal axis) of themovable object. In some instances, an imaging device oriented on themovable object so as to have a field of view oriented upward relative tothe movable object may have an optical axis above a lateral axis (orlongitudinal axis) of the movable object. In some instances, an imagingdevice oriented on the movable object so as to have a field of vieworiented downward relative to the movable object may have an opticalaxis below a lateral axis or longitudinal axis of the movable object.

An imaging device oriented on the movable object so as to have a fieldof view oriented vertically relative to the movable object may have afield of view that is either entirely above a lateral axis (orlongitudinal axis) of the movable object or entirely below a lateralaxis of the movable object. An imaging device oriented on the movableobject so as to have a field of view oriented vertically relative to themovable object may have a field of view that is either substantiallyabove a lateral axis (or longitudinal axis) of the movable object orsubstantially below a lateral axis of the movable object. A field ofview that is substantially above or below a lateral axis (orlongitudinal axis) of the movable object may refer to a field of viewthat is at least 50%, 60%, 70%, 80%, or 90% above or below the lateralaxis or longitudinal axis of the movable object. In some instances, animaging device oriented on the movable object so as to have a field ofview oriented upward relative to the movable object may have a field ofview that is entirely above a lateral axis or longitudinal axis of themovable object. In some instances, an imaging device oriented on themovable object so as to have a field of view oriented upward relative tothe movable object may have a field of view that is substantially abovea lateral axis or longitudinal axis of the movable object. A field ofview that is substantially above a lateral axis or longitudinal axis ofthe movable object may refer to a field of view that is at least 50%,60%, 70%, 80%, or 90% above the lateral axis or longitudinal axis of themovable object. In some instances, an imaging device oriented on themovable object so as to have a field of view oriented downward relativeto the movable object may have a field of view that is entirely below alateral axis or longitudinal axis of the movable object. In someinstances, an imaging device oriented on the movable object so as tohave a field of view oriented downward relative to the movable objectmay have a field of view that is substantially below a lateral axis orlongitudinal axis of the movable object. A field of view that issubstantially below a lateral axis or longitudinal axis of the movableobject may refer to a field of view that is at least 50%, 60%, 70%, 80%,or 90% below the lateral axis or longitudinal axis of the movableobject.

An imaging device that has a field of view that is oriented upward ordownward relative to the movable object may capture image data of anenvironment entirely above the lateral or longitudinal axis of themovable object or image data of an environment entirely below thelateral or longitudinal axis of the movable object. An imaging devicethat has a field of view that is oriented upward or downward relative tothe movable object may capture image data of an environmentsubstantially above the lateral or longitudinal axis of the movableobject or image data of an environment substantially below the lateralor longitudinal axis of the movable object. Image data of an environmentsubstantially above or below the lateral or longitudinal axis of themovable object may refer to image data of an environment at least 50%,60%, 70%, 80%, or 90% above or below the lateral or longitudinal axis ofthe movable object. In some instances, an imaging device that has afield of view oriented upward or downward relative to a movable objectmay capture images that contain image data of an environment that is atleast 50%, 60%, 70%, 80%, 90%, or 100% above a lateral axis (orlongitudinal axis) of the movable object or image data of an environmentthat is at least 50%, 60%, 70%, 80%, 90%, or 100% below a lateral axis(or longitudinal axis) of the movable object.

In some instances, an imaging device that has a field of view orientedupward relative to the movable object may capture image data of anenvironment entirely above the lateral or longitudinal axis of themovable object. In some instances, an imaging device that has a field ofview oriented upward relative to the movable object may capture imagedata of an environment substantially above the lateral or longitudinalaxis of the movable object. Image data of an environment substantiallyabove the lateral or longitudinal axis of the movable object may referto image data of an environment that is at least 50%, 60%, 70%, 80%, or90% above the lateral or longitudinal axis of the movable object. Insome instances, an imaging device that has a field of view orientedupward relative to a movable object may capture images that containimage data of an environment that is at least 50%, 60%, 70%, 80%, 90%,or 100% above a lateral axis (or longitudinal axis) of the movableobject.

In some instances, an imaging device that has a field of view orienteddownward relative to the movable object may capture image data of anenvironment entirely below the lateral or longitudinal axis of themovable object. In some instances, an imaging device that has a field ofview oriented downward relative to the movable object may capture imagedata of an environment substantially below the lateral or longitudinalaxis (or longitudinal axis) of the movable object. Image data of anenvironment substantially below the lateral or longitudinal axis of themovable object may refer to image data of an environment that is atleast 50%, 60%, 70%, 80%, or 90% below the lateral or longitudinal axisof the movable object. In some instances, an imaging device that has afield of view oriented downward relative to a movable object may captureimages that contain image data of an environment that is at least 50%,60%, 70%, 80%, 90%, or 100% below a lateral axis (or longitudinal axis)of the movable object.

While imaging device 102 is shown located above the movable object,systems and methods provided herein may be equally applicable to imagingdevices located below the movable object. The imaging devices disclosedherein may be mounted on an upper surface of the UAV, a lower surface ofthe UAV, on top of or below a central body of the UAV, on top of orbelow a peripheral portion of the UAV, and the like. Thus, an imagingdevice arranged on the movable object so as to have a field of vieworiented vertically relative to the movable object may have a field ofview oriented downward relative to the movable object. While the imagingdevice 102 is shown located substantially near the center of the UAV, itis to be understood that the imaging device may be located anywhere onthe movable object (e.g., near the center, near the edge, etc.). Theimaging device 102 may capture one or more images in an indoor oroutdoor environment. Imaging device 102 may further be used togetherwith one or more processors for navigation, obstacle detection, obstacleavoidance, state determination (e.g., speed measurement), positioning,and the like. The movable object may comprise a plurality of propulsionunits (e.g., 104, 106) configured to affect movement of the movableobject. The propulsion units may comprise a plurality of rotors.

A vertically oriented imaging device may be used to both detectobstacles and determine a state information of the UAV. In someembodiments, a single vertically oriented imaging device may havefunctionalities that cannot be performed by a single laterally orientedimaging device. In some embodiments, imaging devices on the UAV orientedlaterally or sideways may be used to detect obstacles, but may not becapable of determining certain types of UAV state information. Forexample, image data from a single laterally oriented imaging device maynot be useful for determining the position and/or velocity of the UAVwhen the UAV is moving in a direction substantially parallel to theoptical axis of the imaging device. In some embodiments, a singleimaging device oriented vertically (e.g., upwards or downwards) may beused to determine state information of the UAV (e.g., position,velocity) and to detect obstacles, as described further herein. Amulti-functional imaging device may have advantages such as reducedcosts (e.g., versus cost of having two or more imaging devices), loweredweight (e.g., versus weight of having two or more imaging devices) for aUAV, and the like.

FIG. 2 illustrates a schematic of obstacle detection and avoidance, inaccordance with embodiments. FIG. 2 shows a movable object 201 with anupward facing imaging device 202 near an obstacle 206. While FIG. 2depicts an upward facing imaging device, the embodiments describedherein may be equally applicable to downward facing imaging devices. Anobstacle may be located to a side (e.g., lateral side) of the movableobject, e.g., rather than being above or below the movable object, asshown in FIG. 2. Only a part of an obstacle may be located to a lateralside of the movable object as shown by obstacle 206. An obstacle locatedto a lateral side of the movable object may be located in front of themovable object, behind the movable object, to the left of the movableobject, to the right of the movable object, or combinations thereof.

The imaging device may have an angle of view 2θ and a focal length f.When the obstacle is sufficiently close to the imaging device 202, e.g.,within a distance t, the obstacle may occupy a portion of the viewingangle, α. An image captured by the imaging device may have a width 2w ata focal plane, of which part is occupied by the obstacle. The width ofthe image occupied by the obstacle may be referred to herein l, θ, f andw may be known parameters. l may be determined using one or moreprocessors. For example one or more processors may receive an imagecontaining an obstacle. The one or more processors may distinguish(e.g., differentiate) obstacles within the image from other parts (e.g.,distinguish l from 2w−l) via segmentation algorithms. Segmentationalgorithms may divide an image into multiple parts. A segmented imagedata may be used to identify relevant information in digital images. Forexample, images may be segmented using thresholding, clustering methods(e.g., k-means clustering), compression-based methods, histogram-basedmethods, feature detection, edge detection, dual clustering methods,region-growing methods, partial differential equation-based methods,graph partitioning methods, watershed transformations, model basedsegmentation, multi-scale segmentation, semi-automatic segmentation,trainable segmentation, segmentation benchmarking, and the like.Distinguishing an obstacle within the image may comprise detecting anobstacle.

In images in which one or more obstacles are easily distinguished fromthe other parts (e.g., background), color-based image segmentationalgorithms, such as a K-Means clustering algorithm may be used. Forexample, for a movable object operating in an outdoor environment withan imaging device oriented vertically relative to the movable object,captured images may contain obstacles (e.g., a pole) and a background(e.g., the sky). The obstacle may be easily distinguished from thebackground in some embodiments (e.g., blue, clear sky and obstacle ofyellow color) and a color-based image segmentation algorithm may be usedto distinguish the obstacle. In some embodiments, other segmentationmethods, techniques, or algorithms as described herein may be used. Theprocessor may further calculate or measure a width (e.g., l of FIG. 2),length, or area of the obstacle in the image.

In images in which one or more obstacles are not easily distinguishedfrom the other parts (e.g., background), a corner detection algorithmmay be used. For example, a movable object operating in an indoorenvironment coupled to an imaging device may acquire image datacontaining obstacles (e.g., a wall) and a background (e.g., a floor or aceiling). The image data may comprise images of the floor or ceilingmeeting a wall. The obstacle may not easily be distinguished from thebackground in some embodiments (e.g., wall and the ceiling are of thesame white color) and a corner detection algorithm may be used to detectthe boundary between the obstacle and a background. Based on thedetected boundary, the obstacle may be distinguished from thebackground. In some embodiments, other segmentation methods, techniques,or algorithms as described herein may be used. The obstacle may bedistinguished and the processor may further calculate or measure awidth, length, or area of the obstacle.

In FIG. 2, defining

h

to be the height of the obstacle from a position on a horizontal planeof the movable object to the top of the obstacle, the field of view(e.g., angle of view) occupied by the obstacle

(α)

can be calculated according to equations (1) and (2):

$\begin{matrix}{{w - l} = {f*{\tan\left( {\theta - \alpha} \right)}}} & (1) \\{\alpha = {\theta - {\arctan\left( \frac{w - l}{f} \right)}}} & (2)\end{matrix}$

In some embodiments, a distance

t

from the movable object to the obstacle may be determined directly fromthe angle

α

determined in equation (2). A range of possible distances

t

may be determined directly from the angle

α

determined in equation (2). For example, in an indoor environment, apossible range of

h

may be known and small, where

h

is as previously defined. For example, in an indoor environment having afixed height of 3 meters,

h

may range from 3 meters (e.g., UAV is on the floor) to 0 meters (e.g.,UAV is touching the ceiling). Accordingly, a range of possible distancesfrom the movable object to the obstacle may be calculated according tothe equation:t=h*tan(θ−α)   (3)

In some embodiments, a range of possible distances

t

may not be able to be determined directly from the angle

α

determined in equation (2). For example, in an outdoor environment, apossible range of

h

may be unknown. If known, the range of

h

may vary greatly (e.g., between 0 and 500 meters) and

t

calculated based on it may vary greatly.

Alternatively or in addition.

t

may be determined indirectly from the angle

α

as described further below. For example,

t

can be determined based on the changes in the angle

α

as the movable object moves. The movement of the movable object may bealong a known or predetermined path and/or may be of a specified (e.g.,predetermined) distance. In some instances, the movement of the movableobject may be of a specified vertical and/or horizontal distance. Thisapproach can be used to improve the accuracy of the calculation of

t.

FIG. 3 illustrates a movable object 302 with an imaging device 304moving a vertical distance 306, in accordance with embodiments. Theimaging device may capture one or more images at a first location 308.The movable object may travel a vertical distance 306 and the imagingdevice may further capture one or more images at a different location310. Taking equation (3) and dividing both sides by tan (θ−α),

$\begin{matrix}{{{h = {t*{\cot\left( {\theta - \alpha} \right)}}}{dh} = {\frac{t}{\sin^{2}\left( {\theta - \alpha} \right)}d\;\alpha}}{t = {{\sin^{2}\left( {\theta - \alpha} \right)}\frac{d\; h}{d\;\alpha}}}} & (4)\end{matrix}$

t may be obtained using equation (4).

dh

of equation (4) may refer to a vertical distance traveled by the movableobject. For example, with respect to FIG. 3,

dh

may refer to the vertical distance 306 between location 308 and 310. Theheight, or vertical distance traveled by the movable object may bemeasured by a sensor, such as an inertial measurement unit (IMU) or aGPS sensor.

dh

may be about or less than 1 cm, 2 cm, 5 cm, 0.1 m, 0.2 m, 0.5 m, 1 m, 2m, 5 m, or 10 m. In some instances,

dh

may be about 0.2 m to 1 m. The change in the field of view occupied bythe obstacle 307 due to the vertical distance traveled by the movableobject may be defined bydα=α ₁−α₂,whereα₁

and

α₂

are the fields of view occupied by the obstacle each at a locationbefore the vertical movement and after the vertical movement asillustrated in FIG. 3, or

α₁

or

α₂

may be used in equation (4) for

α

and

θ

may be a constant regardless of the UAV location. Multiple

dh

and corresponding

dα

may be obtained (e.g., move the movable object a vertical distancemultiple times), and an average distance of the movable object from theobstacle

(t)

may be obtained for improved accuracy.

In some instances,

dh

may be determined or measured unconditionally each time the movableobject travels a vertical distance. In some instances,

dh

may be determined only if one or more conditions are satisfied. Forexample,

dh

may be determined when a noticeable change of

α (dα)

is detected. In some embodiments, a noticeable change of

α

may be equal to about or less than 5°, 10°, 15°, 20°, 25°, 30°, 35°,40°, or 45°.

FIG. 4 illustrates a movable object 402 with an imaging device 404moving a horizontal distance 406, in accordance with embodiments. Theimaging device may capture one or more images at first location 408. Themovable object may travel a horizontal distance

dt

and the imaging device may capture one or more images at a secondlocation 410. Equation (3) can further be differentiated to become,

$\begin{matrix}{{{d\; t} = {\frac{- h}{\sin^{2}\left( {\theta - \alpha} \right)}d\;\alpha}}{h = {{- {\sin^{2}\left( {\theta - \alpha} \right)}}\frac{d\; t}{d\;\alpha}}}} & (5)\end{matrix}$

h may be obtained using equation (5).

dt

of equation (5) may refer to a horizontal distance traveled by themovable object. For example, with respect to FIG. 4,

dt

may refer to the horizontal distance 406 between location 408 and 410.The horizontal distance traveled by the movable object

(dt)

may be measured by a sensor, such as an inertial measurement unit (IMU)or a GPS sensor.

dt

may be about or less than 1 cm, 2 cm, 5 cm, 0.1 m, 0.2 m, 0.5 m, 1 m, 2m, 5 m, or 10 m. In some instances,

dt

may be about 0.2 m to 1 m. The change in the field of view occupied bythe obstacle due to the horizontal distance traveled by the movableobject may be defined bydα=α ₁−α₂,whereα₁andα₂are the fields of view occupied by the obstacle 407 each at a locationbefore the horizontal movement and after the horizontal movement, asillustrated in FIG. 4.α₁orα₂may be used in equation (5) forandθmay be a constant regardless of the UAV location. Having solved forh,the distance of the movable object from the obstacle(t)may be obtained using equation (3). Multipledtand correspondingdαmay be obtained (e.g., move the movable object a horizontal distancemultiple times), and an average distance of the movable object from theobstacle(t)may be obtained for improved accuracy.

In some instances,

dt

may be determined or measured unconditionally each time the movableobject travels a horizontal distance. In some instances,

dt

may be determined only if one or more conditions are satisfied. Forexample,

dt

may be determined when a noticeable change of

α (dα)

is detected. In some embodiments, a noticeable change of α

may be equal to about or less than 5°, 10°, 15°, 20°, 25°, 30°, 35°,40°, or 45°.

FIG. 5 illustrates a method 500 of detecting obstacles using a movableobject, in accordance with embodiments. In step 502, one or moreprocessors may receive image data from an imaging device carried by themovable object. The image data may comprise obstacle image data andother (e.g., background) image data. For example, for a movable objectoperating in an indoor environment, the image data may comprise imagesof a wall (e.g., an obstacle) and a ceiling (e.g., a background). Forexample, for a movable object operating in an outdoor environment, theimage data may comprise images of an object (e.g., an obstacle) and thesky (e.g., a background). The image data may comprise data of one ormore images captured by the imaging device. For example, the image datamay comprise about 1, 2, 3, 4, 5, 10, 20, 50, 100, 200, 500, 1000, ormore images. The image data may comprise a sequence of images, asdescribed herein. In some embodiments, a processor may comprise afield-programmable gate array (FPGA), application-specific integratedcircuit (ASIC), application-specific standard product (ASSP), or complexprogrammable logic devices (CPLD). In some embodiments, the processormay be an embedded processor carried by the UAV. Alternatively, theprocessor may be separated from the movable object (e.g., at a groundstation, communicating with the movable object). The imaging device maybe arranged on the movable object so as to have a field of view orientedvertically relative to the movable object (e.g., upwards or downwards),as mentioned herein. The obstacle may be located to a side (e.g.,lateral side) of the movable object, e.g., rather than being above orbelow the movable object. For example, the obstacle may be located infront of the movable object, behind the movable object, to the left ofthe movable object, to the right of the movable object, or combinationsthereof. For example, at least a part of the obstacle may be located ona lateral side of the UAV. Thus, in some embodiments, the image datareceived by processors in step 502 may comprise image data of anobstacle positioned at a lateral side of the movable object.

In step 504, the one or more processors may calculate a distance betweenthe movable object and the obstacle. The calculation may be based on theimage data of the obstacle. Calculating a distance between the movableobject and the obstacle may comprise detecting an obstacle. Calculatinga distance between the movable object and the obstacle may comprisedifferentiating between a portion of the image data depicting thebackground and a portion of the image data depicting the obstacle.Calculating a distance between the movable object and the obstacle maycomprise segmenting the image data. Segmenting an image data maycomprise threshold methods (e.g., Otsu's method), color-basedsegmentation (e.g., K-means clustering), transform methods (e.g.,watershed segmentation), texture methods (e.g., texture filters), andthe like as mentioned herein.

There may be a variety of ways to calculate a distance between themovable object and the obstacle, as further described below. Thecalculation may depend on external factors, such as the environment type(e.g., indoor, outdoor, etc.) in which the movable object is operatingin. The distance from the movable object to the obstacle may beestimated based on an angle of view of the obstacle within the imagedata. In some embodiments, a negative angle value may be applied in theestimation. As used herein, an angle of view may have a negative anglevalue when an imaging device has an optical axis below the horizontalaxis.

Based on detection of the obstacle or the calculation of step 504, oneor more processors may initiate an obstacle avoidance program and/orgenerate a warning signal. For example, as soon as an obstacle isdetected, an obstacle avoidance program may be initiated and/or awarning signal may be generated. For example, if the distance betweenthe movable object and the obstacle is calculated to be about 0.1 m, 0.2m, 0.3 m, 0.4 m, 0.5 m, 0.75 m, 1 m, 1.25 m, 1.5 m, 2 m, 2.5 m, 3 m, 4m, 5 m, 10 m, 15 m, 20 m, 50 m, 100 m, 150 m, or more, an obstacleavoidance program may be initiated and/or a warning signal may begenerated. The obstacle avoidance program may involve disallowing themovable object to proceed in the direction of the obstacle. The obstacleavoidance program may involve moving the movable object in an oppositedirection of the detected obstacle (e.g., via control signals output topropulsion units of the movable object). The obstacle avoidance programmay involve generating and outputting control signals to one or morepropulsion units of the movable object to move the movable object (e.g.,vertical or lateral movements) such that the obstacle is avoided.Warning signals may involve outputting haptic, visual, or auditorywarnings to an operator or a user. For example, the movable object orcomponents coupled with the movable object may output flashing lights tosignal a warning. For example, a visual warning may be displayed on aremote control of an operator to signal a warning. For example, a soundmay be generated from the movable object, a remote control, or any othercomponents coupled with the movable object to signal a warning. Forexample, the UAV, a remote control, or any other components coupled withthe movable object may vibrate to signal a warning.

While imaging devices arranged on a movable object so as to have a fieldof view oriented vertically relative to the movable object and anobstacle located to a lateral side of the movable object has beendiscussed primarily herein, the methods and systems may be equallyapplicable to any arrangement in which an optical axis of the imagingdevice relative to an obstacle and a location of the imaging devicerelative to the obstacle is substantially similar to the arrangementdiscussed herein. For example, the methods and systems may be applied toimaging devices arranged on a movable object so as to have a field ofview oriented horizontally relative to the movable object and anobstacle located above or below the movable object. This may includewhen an optical axis is substantially directed in a first direction andthe obstacle is in a second direction from the imaging devicesubstantially perpendicular to the first direction.

FIG. 6 illustrates a method 600 of detecting obstacles using an imagingdevice carried by a movable object, in accordance with embodiments. Instep 602, one or more processors may receive a first image data of anobstacle from the imaging device carried by the movable object,substantially as described herein. The imaging device may be arranged onthe movable object so as to have a field of view oriented upward ordownward (e.g., vertically) relative to the movable object.

In step 604, the one or more processors may generate control signals forcontrolling one or more propulsion units of the movable object to causethe movable object to move. In some instances, the movement of themovable object may be in a horizontal direction, a vertical direction,or any combination thereof. The horizontal and/or vertical movement maybe of any combination and/or sequence. The movable object may move in apredetermined path or sequence which may include any horizontal and/orvertical distance. The movable object may move without a predeterminedpath or sequence in any direction (e.g., horizontal and/or verticaldirection) and in any distance (e.g., horizontal and/or verticaldirection). Alternatively or in conjunction, the movable object mayprocess the first image data and based on information obtained (e.g.,obstacle detected), move in a certain direction (e.g., horizontal and/orvertical direction). Movement of the UAV may cause the UAV to move acertain horizontal and/or vertical distance. The horizontal distancethat is moved may be about 0.01 m, 0.05 m, 0.1 m, 0.2 m, 0.5 m, 0.75 m,1 m, 1.5 m, 2 m, or more. The vertical distance that is moved may beabout 0.01 m, 0.05 m, 0.1 m, 0.2 m, 0.5 m, 0.75 m, 1 m, 1.5 m, 2 m, ormore. The horizontal distance and/or vertical distance moved may bemeasured and stored in a processor. For example, the distance themovable object moves may be measured by an inertial measurement unit andthe measurement may be stored in the one or more processors.

In step 606, the one or more processors may receive a second image dataof the obstacle from the imaging device after the movable object hasmoved. In step 608, the one or more processors may calculate a distancebetween the movable object and the obstacle based on the first imagedata and the second image data, substantially as described herein. Thecalculating step may comprise assessing the first and second image datain order to detect changes in a portion of the field of view of theimaging device occupied by the obstacle prior to and after the movableobject has moved the horizontal and/or vertical distance.

Method 600 may be repeated as desired. A plurality of distances betweenthe movable object and the obstacle may be obtained. Statisticalanalysis may be performed on the plurality of distances obtained betweenthe movable object and the obstacle for improved accuracy. For example,the plurality of distances obtained between the movable object and theobstacle may be averaged for improved accuracy. Method 600 may berepeated about or more than 1, 2, 3, 4, 5, 10, or 20 times beforedetermining a distance between the movable object and the obstacle.

FIG. 7 illustrates a method 700 for determining state information for amovable object, in accordance with embodiments. In step 702, one or moreprocessors may receive image data from an imaging device carried by themovable object. The imaging device may be arranged on the movable objectso as to have a field of view oriented upwards relative to the movableobject, substantially as described herein. An upward facing camera maycapture images including a background. For example, the background maycomprise a ceiling (e.g., in an indoor environment) or the sky (e.g., inan outdoor environment). The obstacle may be easily distinguished fromthe background in some embodiments (e.g., blue, clear sky and obstacleof different color) and segmentation algorithm may be used todistinguish the obstacle as described herein. Thus, an upward facingcamera may advantageously capture images which may be likely to havebackgrounds of lower complexity than other orientations, which mayincrease the ease with which an obstacle may be distinguished from thebackground. This may result in less processing power being required toanalyze the captured image.

In step 704, the one or more processors may receive attitude informationfor the movable object. The attitude information may comprise a rollorientation, a pitch orientation, or a yaw orientation of the movableobject.

In step 706, the one or more processors may calculate at least one ofposition information or motion information for the movable object basedon the image data and the attitude information for the movable object.For example, if the movable object is tilted, such that the verticalaxis of the UAV is not perpendicular to the straight line distance tothe obstacle, the one or more processors may need to account for theangle of tilt in the position and/or motion information calculations. Insome instances, the attitude information for the movable object may beused to accurately calculate or determine state information (e.g.,position or motion information) of the movable object. For example, whena UAV has a roll or pitch orientation, an image taken (e.g., captured)by the imaging device may be a transformed version (e.g., via affinetransform) of an image captured by an imaging device on a UAV that hasno roll or pitch orientation. In such a case, a coordinate rotation ofthe image may be effected based on the attitude information for themovable object. Assuming roll and pitch angles of

ψ and φ,

a corresponding rotation matrix

R

can be formulated as shown below. In some instances, a yaw angle is notimportant and may be ignored.

$R = {\begin{pmatrix}{\cos\mspace{11mu}\phi} & 0 & {\sin\mspace{11mu}\phi} \\0 & 1 & 0 \\{{- \sin}\mspace{11mu}\phi} & 0 & {\cos\mspace{11mu}\phi}\end{pmatrix}\begin{pmatrix}1 & 0 & 0 \\0 & {\cos\mspace{11mu}\psi} & {{- \sin}\mspace{11mu}\psi} \\0 & {\sin\mspace{11mu}\psi} & {\cos\mspace{11mu}\psi}\end{pmatrix}}$

For pixel (u,v) on the image, a 3D point

(x,y,z)

in focal plane corresponding to the pixel may be calculated as follows:

$\begin{pmatrix}x \\y \\z\end{pmatrix} = {K^{- 1}\begin{pmatrix}u \\v \\1\end{pmatrix}}$

The 3D point may further be rotated according to rotation matrix:

$\begin{pmatrix}x^{\prime} \\y^{\prime} \\z^{\prime}\end{pmatrix} = {R\begin{pmatrix}x \\y \\z\end{pmatrix}}$

The rotated 3D point may be projected onto the image plane according tothe following which may be used to more accurately determine a positionor motion information of the UAV:

$\begin{pmatrix}u^{\prime} \\v^{\prime} \\1\end{pmatrix} = {{K\begin{pmatrix}x^{\prime} \\y^{\prime} \\z^{\prime}\end{pmatrix}} = {{KRK}^{- 1}\begin{pmatrix}u \\v \\1\end{pmatrix}}}$

The motion information may comprise a velocity of the movable object.For example, the one or more processors may receive a sequence of two ormore images. The processors may analyze the sequence of images anddetermine a change. For example, the processor may detect a reproduciblefeature within the image and determine its relative position within theimage over the sequence of images. The processor may compare therelative position change against a time lapse between the sequence ofimages to determine a speed of the movable object. Step 706 may comprisedetecting feature points in each image of the sequence of images anddetermining changes in the feature points over the sequence of images. Afeature point can be a portion of an image (e.g., an edge, corner,interest point, blob, ridge, etc.) that is uniquely distinguishable fromthe remaining portions of the image and/or other feature points in theimage. Optionally, a feature point may be relatively invariant totransformations of the imaged object (e.g., translation, rotation,scaling) and/or changes in the characteristics of the image (e.g.,brightness, exposure). A feature point may be detected in portions of animage that is rich in terms of informational content (e.g., significant2D texture). A feature point may be detected in portions of an imagethat are stable under perturbations (e.g., when varying illumination andbrightness of an image).

Feature detection as described herein can be accomplished using variousalgorithms which may extract one or more feature points from image data.The algorithm may be an edge detection algorithm, a corner detectionalgorithm, a blob detection algorithm, or a ridge detection algorithm.In some embodiments, the corner detection algorithm may be a “Featuresfrom accelerated segment test” (FAST). In some embodiments, the featuredetector may extract feature points and make calculations regardingfeature points using FAST. In some embodiments, the feature detector canbe a Canny edge detector, Sobel operator, Harris &Stephens/Plessy/Shi-Tomasi corner detection algorithm, the SUSAN cornerdetector, Level curve curvature approach, Laplacian of Gaussian,Difference of Gaussians, Determinant of Hessian, MSER, PCBR, orGrey-level blobs, ORB, FREAK, or suitable combinations thereof.

Step 706 may involve calculating a position information for the movableobject. An environmental map may be generated using the movable object.For example, the environmental map may be generated using SimultaneousLocalization and Mapping (SLAM) techniques. For example, in a limitedindoor environment, the one or more imaging devices may acquire imagesof the various areas of the environment. The one or more processors mayfurther process the image data and create a global map of theenvironment. The position of the movable object may be calculatedrelative to the generated map of the environment.

Based on at least one of the calculated position information or motioninformation, control signals may be output to control the position ormotion of the movable object. For example, if the movable object isdetermined to be going above some threshold velocity, control signalsmay be output to reduce the velocity of the movable object. For example,if the movable object is determined to be going below some thresholdvelocity, control signals may be output to increase the velocity of themovable object.

FIG. 8 illustrates a method of detecting obstacles using a movableobject, in accordance with embodiments. In step 802, one or moreprocessors may receive image data of an obstacle from an imaging devicecarried by the movable object, substantially as described herein. Theimaging device may be arranged on the movable object so as to have afield of view oriented upwards relative to the movable object.

In step 804, one or more processors may receive an attitude informationfor the movable objects. The attitude information may be generated by asensor, such as an inertial measurement unit (IMU). The attitude maycomprise a roll orientation, a pitch orientation, or a yaw orientationof the vehicle.

In step 806, the one or more processors may calculate a distance betweenthe movable object and the obstacle based on the image data of theobstacle and the attitude information for the movable object,substantially as described herein. For example, if the movable object istilted, such that the vertical axis of the UAV is not perpendicular tothe straight line distance to the obstacle, the one or more processorsmay need to account for the angle of tilt in the obstacle distancecalculations.

In some instances, the attitude information for the movable object maybe used to accurately calculate or determine a distance between themovable object and the obstacle. For example, when a UAV has a roll orpitch orientation, an image taken (e.g., captured) by the imaging devicemay be a transformed version (e.g., via affine transform) of an imagecaptured by an imaging device on a UAV that has no roll or pitchorientation. In such a case, a coordinate rotation of the image may beeffected based on the attitude information of the movable object.Assuming roll and pitch angles of

ψ

and

φ,

a corresponding rotation matrix

R

can be formulated as shown below. In some instances, a yaw angle is notimportant and may be ignored.

$R = {\begin{pmatrix}{\cos\mspace{11mu}\phi} & 0 & {\sin\mspace{11mu}\phi} \\0 & 1 & 0 \\{{- \sin}\mspace{11mu}\phi} & 0 & {\cos\mspace{11mu}\phi}\end{pmatrix}\begin{pmatrix}1 & 0 & 0 \\0 & {\cos\mspace{11mu}\psi} & {{- \sin}\mspace{11mu}\psi} \\0 & {\sin\mspace{11mu}\psi} & {\cos\mspace{11mu}\psi}\end{pmatrix}}$

For pixel (u,v) on the image, a 3D point

(x,y,z)

in focal plane corresponding to the pixel may be calculated as follows:

$\begin{pmatrix}x \\y \\z\end{pmatrix} = {K^{- 1}\begin{pmatrix}u \\v \\1\end{pmatrix}}$

The 3D point may further be rotated according to rotation matrix:

$\begin{pmatrix}x^{\prime} \\y^{\prime} \\z^{\prime}\end{pmatrix} = {R\begin{pmatrix}x \\y \\z\end{pmatrix}}$

The rotated 3D point may be projected onto the image plane according tothe following which may be used to more accurately determine a distancebetween a movable object and an obstacle:

$\begin{pmatrix}u^{\prime} \\v^{\prime} \\1\end{pmatrix} = {{K\begin{pmatrix}x^{\prime} \\y^{\prime} \\z^{\prime}\end{pmatrix}} = {{KRK}^{- 1}\begin{pmatrix}u \\v \\1\end{pmatrix}}}$

FIG. 9 illustrates a UAV 902 operating in an outdoor environment 900, inaccordance with embodiments. The outdoor environment 900 may be anurban, suburban, or rural setting, or any other environment that is notat least partially within a building. The UAV 902 may be operatedrelatively close to the ground 904 (e.g., low altitude) or relativelyfar from the ground 904 (e.g., high altitude). For example, a UAV 902operating less than or equal to approximately 10 m from the ground maybe considered to be at low altitude, while a UAV 902 operating atgreater than or equal to approximately 10 m from the ground may beconsidered to be at high altitude.

In some embodiments, the outdoor environment 900 includes one or moreobstacles 908 a-d. An obstacle may include any object or entity that mayobstruct the movement of the UAV 902. Some obstacles may be situated onthe ground 904 (e.g., obstacles 908 a, 908 d), such as buildings, groundvehicles (e.g., cars, motorcycles, trucks, bicycles), human beings,animals, plants (e.g., trees, bushes), and other manmade or naturalstructures. Some obstacles may be in contact with and/or supported bythe ground 904, water, manmade structures, or natural structures.Alternatively, some obstacles may be wholly located in the air 906(e.g., obstacles 908 b, 908 c), including aerial vehicles (e.g.,airplanes, helicopters, hot air balloons, other UAVs) or birds. Aerialobstacles may not be supported by the ground 904, or by water, or by anynatural or manmade structures. An obstacle located on the ground 904 mayinclude portions that extend substantially into the air 906 (e.g., tallstructures such as towers, skyscrapers, lamp posts, radio towers, powerlines, trees, etc.). A UAV operating in an outdoor environment maycapture images of said obstacles. For example, an imaging deviceoriented vertically relative to the UAV as described herein may captureimages of the obstacles. The captured images may be processed by aprocessor to be used in obstacle detection and avoidance, navigation,and/or state determination of the UAV.

FIG. 10 illustrates a UAV 1052 operating in an indoor environment 1050,in accordance with embodiments. The indoor environment 1050 is withinthe interior of a building 1054 having a floor 1056, one or more walls1058, and/or a ceiling or roof 1060. Exemplary buildings includeresidential, commercial, or industrial buildings such as houses,apartments, offices, manufacturing facilities, storage facilities, andso on. The interior of the building 1054 may be completely enclosed bythe floor 1056, walls 1058, and ceiling 1060 such that the UAV 1052 isconstrained to the interior space. Conversely, at least one of the floor1056, walls 1058, or ceiling 1060 may be absent, thereby enabling theUAV 1052 to fly from inside to outside, or vice-versa. Alternatively orin combination, one or more apertures 1064 may be formed in the floor1056, walls 1058, or ceiling 1060 (e.g., a door, window, skylight).

Similar to the outdoor environment 1000, the indoor environment 1050 caninclude one or more obstacles 1062 a-d. Some obstacles may be situatedon the floor 1056 (e.g., obstacle 1062 a), such as furniture,appliances, human beings, animals, plants, and other manmade or naturalobjects. Conversely, some obstacles may be located in the air (e.g.,obstacle 1062 b), such as birds or other UAVs. Some obstacles in theindoor environment 1050 can be supported by other structures or objects.Obstacles may also be attached to the ceiling 1060 (e.g., obstacle 1062c), such as light fixtures, ceiling fans, beams, or otherceiling-mounted appliances or structures. In some embodiments, obstaclesmay be attached to the walls 1058 (e.g., obstacle 1062 d), such as lightfixtures, shelves, cabinets, and other wall-mounted appliances orstructures. Notably, the structural components of the building 1054 canalso be considered to be obstacles, including the floor 1056, walls1058, and ceiling 1060. A UAV operating in an outdoor environment maycapture images of said obstacles. For example, an imaging deviceoriented vertically relative to the UAV as described herein may captureimages of the obstacles. The captured images may be processed by aprocessor to be used in obstacle detection and avoidance, navigation,and/or state determination of the UAV.

The obstacles described herein may be substantially stationary (e.g.,buildings, plants, structures) or substantially mobile (e.g., humanbeings, animals, vehicles, or other objects capable of movement). Someobstacles may include a combination of stationary and mobile components(e.g., a windmill). Mobile obstacles or obstacle components may moveaccording to a predetermined or predictable path or pattern. Forexample, the movement of a car may be relatively predictable (e.g.,according to the shape of the road). Alternatively, some mobileobstacles or obstacle components may move along random or otherwiseunpredictable trajectories. For example, a living being such as ananimal may move in a relatively unpredictable manner.

The embodiments provided herein can be applied to various types of UAVs.For instance, the UAV may be a small-scale UAV that weighs no more than10 kg and/or has a maximum dimension of no more than 1.5 m. In someembodiments, the UAV may be a rotorcraft, such as a multi-rotor aircraftthat is propelled to move through the air by a plurality of propellers(e.g., a quadcopter). Additional examples of UAVs and other movableobjects suitable for use with the embodiments presented herein aredescribed in further detail below.

The UAVs described herein can be operated completely autonomously (e.g.,by a suitable computing system such as an onboard controller),semi-autonomously, or manually (e.g., by a human user). The UAV canreceive commands from a suitable entity (e.g., human user or autonomouscontrol system) and respond to such commands by performing one or moreactions. For example, the UAV can be controlled to take off from theground, move within the air (e.g., with up to three degrees of freedomin translation and up to three degrees of freedom in rotation), move totarget location or to a sequence of target locations, hover within theair, land on the ground, and so on. As another example, the UAV can becontrolled to move at a specified velocity and/or acceleration (e.g.,with up to three degrees of freedom in translation and up to threedegrees of freedom in rotation) or along a specified movement path.Furthermore, the commands can be used to control one or more UAVcomponents, such as the components described herein (e.g., sensors,actuators, propulsion units, payload, etc.). For instance, some commandscan be used to control the position, orientation, and/or operation of aUAV payload such as a camera. Optionally, the UAV can be configured tooperate in accordance with one or more predetermined operating rules.The operating rules may be used to control any suitable aspect of theUAV, such as the position (e.g., latitude, longitude, altitude),orientation (e.g., roll, pitch yaw), velocity (e.g., translationaland/or angular), and/or acceleration (e.g., translational and/orangular) of the UAV. For instance, the operating rules can be designedsuch that the UAV is not permitted to fly beyond a threshold height,e.g., the UAV can be configured to fly at a height of no more than 400 mfrom the ground. In some embodiments, the operating rules can be adaptedto provide automated mechanisms for improving UAV safety and preventingsafety incidents. For example, the UAV can be configured to detect arestricted flight region (e.g., an airport) and not fly within apredetermined distance of the restricted flight region, thereby avertingpotential collisions with aircraft and other obstacles.

The systems, devices, and methods described herein can be applied to awide variety of movable objects. As previously mentioned, anydescription herein of a UAV may apply to and be used for any movableobject. A movable object of the present invention can be configured tomove within any suitable environment, such as in air (e.g., a fixed-wingaircraft, a rotary-wing aircraft, or an aircraft having neither fixedwings nor rotary wings), in water (e.g., a ship or a submarine), onground (e.g., a motor vehicle, such as a car, truck, bus, van,motorcycle; a movable structure or frame such as a stick, fishing pole;or a train), under the ground (e.g., a subway), in space (e.g., aspaceplane, a satellite, or a probe), or any combination of theseenvironments. The movable object can be a vehicle, such as a vehicledescribed elsewhere herein. In some embodiments, the movable object canbe mounted on a living subject, such as a human or an animal. Suitableanimals can include avians, canines, felines, equines, bovines, ovines,porcines, delphines, rodents, or insects.

The movable object may be capable of moving freely within theenvironment with respect to six degrees of freedom (e.g., three degreesof freedom in translation and three degrees of freedom in rotation).Alternatively, the movement of the movable object can be constrainedwith respect to one or more degrees of freedom, such as by apredetermined path, track, or orientation. The movement can be actuatedby any suitable actuation mechanism, such as an engine or a motor. Theactuation mechanism of the movable object can be powered by any suitableenergy source, such as electrical energy, magnetic energy, solar energy,wind energy, gravitational energy, chemical energy, nuclear energy, orany suitable combination thereof. The movable object may beself-propelled via a propulsion system, as described elsewhere herein.The propulsion system may optionally run on an energy source, such aselectrical energy, magnetic energy, solar energy, wind energy,gravitational energy, chemical energy, nuclear energy, or any suitablecombination thereof. Alternatively, the movable object may be carried bya living being.

In some instances, the movable object can be a vehicle. Suitablevehicles may include water vehicles, aerial vehicles, space vehicles, orground vehicles. For example, aerial vehicles may be fixed-wing aircraft(e.g., airplane, gliders), rotary-wing aircraft (e.g., helicopters,rotorcraft), aircraft having both fixed wings and rotary wings, oraircraft having neither (e.g., blimps, hot air balloons). A vehicle canbe self-propelled, such as self-propelled through the air, on or inwater, in space, or on or under the ground. A self-propelled vehicle canutilize a propulsion system, such as a propulsion system including oneor more engines, motors, wheels, axles, magnets, rotors, propellers,blades, nozzles, or any suitable combination thereof. In some instances,the propulsion system can be used to enable the movable object to takeoff from a surface, land on a surface, maintain its current positionand/or orientation (e.g., hover), change orientation, and/or changeposition.

The movable object can be controlled remotely by a user or controlledlocally by an occupant within or on the movable object. In someembodiments, the movable object is an unmanned movable object, such as aUAV. An unmanned movable object, such as a UAV, may not have an occupantonboard the movable object. The movable object can be controlled by ahuman or an autonomous control system (e.g., a computer control system),or any suitable combination thereof. The movable object can be anautonomous or semi-autonomous robot, such as a robot configured with anartificial intelligence.

The movable object can have any suitable size and/or dimensions. In someembodiments, the movable object may be of a size and/or dimensions tohave a human occupant within or on the vehicle. Alternatively, themovable object may be of size and/or dimensions smaller than thatcapable of having a human occupant within or on the vehicle. The movableobject may be of a size and/or dimensions suitable for being lifted orcarried by a human. Alternatively, the movable object may be larger thana size and/or dimensions suitable for being lifted or carried by ahuman. In some instances, the movable object may have a maximumdimension (e.g., length, width, height, diameter, diagonal) of less thanor equal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. Themaximum dimension may be greater than or equal to about: 2 cm, 5 cm, 10cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. For example, the distance betweenshafts of opposite rotors of the movable object may be less than orequal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m.Alternatively, the distance between shafts of opposite rotors may begreater than or equal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m,or 10 m.

In some embodiments, the movable object may have a volume of less than100 cm×100 cm×100 cm, less than 50 cm×50 cm×30 cm, or less than 5 cm×5cm×3 cm. The total volume of the movable object may be less than orequal to about: 1 cm³, 2 cm³, 5 cm³, 10 cm³, 20 cm³, 30 cm³, 40 cm³, 50cm³, 60 cm³, 70 cm³, 80 cm³, 90 cm³, 100 cm³, 150 cm³, 200 cm³, 300 cm³,500 cm³, 750 cm³, 1000 cm³, 5000 cm³, 10,000 cm³, 100,000 cm³, 1 m³, or10 m³. Conversely, the total volume of the movable object may be greaterthan or equal to about: 1 cm³, 2 cm³, 5 cm³, 10 cm³, 20 cm³, 30 cm³, 40cm³, 50 cm³, 60 cm³, 70 cm³, 80 cm³, 90 cm³, 100 cm³, 150 cm³, 200 cm³,300 cm³, 500 cm³, 750 cm³, 1000 cm³, 5000 cm³, 10,000 cm³, 100,000 cm³,1 m³, or 10 m³.

In some embodiments, the movable object may have a footprint (which mayrefer to the lateral cross-sectional area encompassed by the movableobject) less than or equal to about: 32,000 cm², 20,000 cm², 10,000 cm²,1,000 cm², 500 cm², 100 cm², 50 cm², 10 cm², or 5 cm². Conversely, thefootprint may be greater than or equal to about: 32,000 cm², 20,000 cm²,10,000 cm², 1,000 cm², 500 cm², 100 cm², 50 cm², 10 cm², or 5 cm².

In some instances, the movable object may weigh no more than 1000 kg.The weight of the movable object may be less than or equal to about:1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg,8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg,or 0.01 kg. Conversely, the weight may be greater than or equal toabout: 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10kg, 9 kg, 8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1kg, 0.05 kg, or 0.01 kg.

In some embodiments, a movable object may be small relative to a loadcarried by the movable object. The load may include a payload and/or acarrier, as described in further detail below. In some examples, a ratioof a movable object weight to a load weight may be greater than, lessthan, or equal to about 1:1. In some instances, a ratio of a movableobject weight to a load weight may be greater than, less than, or equalto about 1:1. Optionally, a ratio of a carrier weight to a load weightmay be greater than, less than, or equal to about 1:1. When desired, theratio of an movable object weight to a load weight may be less than orequal to: 1:2, 1:3, 1:4, 1:5, 1:10, or even less. Conversely, the ratioof a movable object weight to a load weight can also be greater than orequal to: 2:1, 3:1, 4:1, 5:1, 10:1, or even greater.

In some embodiments, the movable object may have low energy consumption.For example, the movable object may use less than about: 5 W/h, 4 W/h, 3W/h, 2 W/h, 1 W/h, or less. In some instances, a carrier of the movableobject may have low energy consumption. For example, the carrier may useless than about: 5 W/h, 4 W/h, 3 W/h, 2 W/h, 1 W/h, or less. Optionally,a payload of the movable object may have low energy consumption, such asless than about: 5 W/h, 4 W/h, 3 W/h, 2 W/h, 1 W/h, or less.

FIG. 11 illustrates an unmanned aerial vehicle (UAV) 1100, in accordancewith embodiments of the present invention. The UAV may be an example ofa movable object as described herein. The UAV 1100 can include apropulsion system having four rotors 1102, 1104, 1106, and 1108. Anynumber of rotors may be provided (e.g., one, two, three, four, five,six, seven, eight, or more). The rotors, rotor assemblies, or otherpropulsion systems of the unmanned aerial vehicle may enable theunmanned aerial vehicle to hover/maintain position, change orientation,and/or change location. The distance between shafts of opposite rotorscan be any suitable length 1110. For example, the length 1110 can beless than or equal to 2 m, or less than equal to 5 m. In someembodiments, the length 1110 can be within a range from 40 cm to 1 m,from 10 cm to 2 m, or from 5 cm to 5 m. Any description herein of a UAVmay apply to a movable object, such as a movable object of a differenttype, and vice versa.

In some embodiments, the movable object can be configured to carry aload. The load can include one or more of passengers, cargo, equipment,instruments, and the like. The load can be provided within a housing.The housing may be separate from a housing of the movable object, or bepart of a housing for a movable object. Alternatively, the load can beprovided with a housing while the movable object does not have ahousing. Alternatively, portions of the load or the entire load can beprovided without a housing. The load can be rigidly fixed relative tothe movable object. Optionally, the load can be movable relative to themovable object (e.g., translatable or rotatable relative to the movableobject).

In some embodiments, the load includes a payload. The payload can beconfigured not to perform any operation or function. Alternatively, thepayload can be a payload configured to perform an operation or function,also known as a functional payload. For example, the payload can includeone or more sensors for surveying one or more targets. Any suitablesensor can be incorporated into the payload, such as an image capturedevice (e.g., a camera), an audio capture device (e.g., a parabolicmicrophone), an infrared imaging device, or an ultraviolet imagingdevice. The sensor can provide static sensing data (e.g., a photograph)or dynamic sensing data (e.g., a video). In some embodiments, the sensorprovides sensing data for the target of the payload. Alternatively or incombination, the payload can include one or more emitters for providingsignals to one or more targets. Any suitable emitter can be used, suchas an illumination source or a sound source. In some embodiments, thepayload includes one or more transceivers, such as for communicationwith a module remote from the movable object. Optionally, the payloadcan be configured to interact with the environment or a target. Forexample, the payload can include a tool, instrument, or mechanismcapable of manipulating objects, such as a robotic arm.

Optionally, the load may include a carrier. The carrier can be providedfor the payload and the payload can be coupled to the movable object viathe carrier, either directly (e.g., directly contacting the movableobject) or indirectly (e.g., not contacting the movable object).Conversely, the payload can be mounted on the movable object withoutrequiring a carrier. The payload can be integrally formed with thecarrier. Alternatively, the payload can be releasably coupled to thecarrier. In some embodiments, the payload can include one or morepayload elements, and one or more of the payload elements can be movablerelative to the movable object and/or the carrier, as described above.

The carrier can be integrally formed with the movable object.Alternatively, the carrier can be releasably coupled to the movableobject. The carrier can be coupled to the movable object directly orindirectly. The carrier can provide support to the payload (e.g., carryat least part of the weight of the payload). The carrier can include asuitable mounting structure (e.g., a gimbal platform) capable ofstabilizing and/or directing the movement of the payload. In someembodiments, the carrier can be adapted to control the state of thepayload (e.g., position and/or orientation) relative to the movableobject. For example, the carrier can be configured to move relative tothe movable object (e.g., with respect to one, two, or three degrees oftranslation and/or one, two, or three degrees of rotation) such that thepayload maintains its position and/or orientation relative to a suitablereference frame regardless of the movement of the movable object. Thereference frame can be a fixed reference frame (e.g., the surroundingenvironment). Alternatively, the reference frame can be a movingreference frame (e.g., the movable object, a payload target).

In some embodiments, the carrier can be configured to permit movement ofthe payload relative to the carrier and/or movable object. The movementcan be a translation with respect to up to three degrees of freedom(e.g., along one, two, or three axes) or a rotation with respect to upto three degrees of freedom (e.g., about one, two, or three axes), orany suitable combination thereof.

In some instances, the carrier can include a carrier frame assembly anda carrier actuation assembly. The carrier frame assembly can providestructural support to the payload. The carrier frame assembly caninclude individual carrier frame components, some of which can bemovable relative to one another. The carrier actuation assembly caninclude one or more actuators (e.g., motors) that actuate movement ofthe individual carrier frame components. The actuators can permit themovement of multiple carrier frame components simultaneously, or may beconfigured to permit the movement of a single carrier frame component ata time. The movement of the carrier frame components can produce acorresponding movement of the payload. For example, the carrieractuation assembly can actuate a rotation of one or more carrier framecomponents about one or more axes of rotation (e.g., roll axis, pitchaxis, or yaw axis). The rotation of the one or more carrier framecomponents can cause a payload to rotate about one or more axes ofrotation relative to the movable object. Alternatively or incombination, the carrier actuation assembly can actuate a translation ofone or more carrier frame components along one or more axes oftranslation, and thereby produce a translation of the payload along oneor more corresponding axes relative to the movable object.

In some embodiments, the movement of the movable object, carrier, andpayload relative to a fixed reference frame (e.g., the surroundingenvironment) and/or to each other, can be controlled by a terminal. Theterminal can be a remote control device at a location distant from themovable object, carrier, and/or payload. The terminal can be disposed onor affixed to a support platform. Alternatively, the terminal can be ahandheld or wearable device. For example, the terminal can include asmartphone, tablet, laptop, computer, glasses, gloves, helmet,microphone, or suitable combinations thereof. The terminal can include auser interface, such as a keyboard, mouse, joystick, touchscreen, ordisplay. Any suitable user input can be used to interact with theterminal, such as manually entered commands, voice control, gesturecontrol, or position control (e.g., via a movement, location or tilt ofthe terminal).

The terminal can be used to control any suitable state of the movableobject, carrier, and/or payload. For example, the terminal can be usedto control the position and/or orientation of the movable object,carrier, and/or payload relative to a fixed reference from and/or toeach other. In some embodiments, the terminal can be used to controlindividual elements of the movable object, carrier, and/or payload, suchas the actuation assembly of the carrier, a sensor of the payload, or anemitter of the payload. The terminal can include a wirelesscommunication device adapted to communicate with one or more of themovable object, carrier, or payload.

The terminal can include a suitable display unit for viewing informationof the movable object, carrier, and/or payload. For example, theterminal can be configured to display information of the movable object,carrier, and/or payload with respect to position, translationalvelocity, translational acceleration, orientation, angular velocity,angular acceleration, or any suitable combinations thereof. In someembodiments, the terminal can display information provided by thepayload, such as data provided by a functional payload (e.g., imagesrecorded by a camera or other image capturing device).

Optionally, the same terminal may both control the movable object,carrier, and/or payload, or a state of the movable object, carrierand/or payload, as well as receive and/or display information from themovable object, carrier and/or payload. For example, a terminal maycontrol the positioning of the payload relative to an environment, whiledisplaying image data captured by the payload, or information about theposition of the payload. Alternatively, different terminals may be usedfor different functions. For example, a first terminal may controlmovement or a state of the movable object, carrier, and/or payload whilea second terminal may receive and/or display information from themovable object, carrier, and/or payload. For example, a first terminalmay be used to control the positioning of the payload relative to anenvironment while a second terminal displays image data captured by thepayload. Various communication modes may be utilized between a movableobject and an integrated terminal that both controls the movable objectand receives data, or between the movable object and multiple terminalsthat both control the movable object and receives data. For example, atleast two different communication modes may be formed between themovable object and the terminal that both controls the movable objectand receives data from the movable object.

FIG. 12 illustrates a movable object 1200 including a carrier 1202 and apayload 1204, in accordance with embodiments. Although the movableobject 1200 is depicted as an aircraft, this depiction is not intendedto be limiting, and any suitable type of movable object can be used, aspreviously described herein. One of skill in the art would appreciatethat any of the embodiments described herein in the context of aircraftsystems can be applied to any suitable movable object (e.g., an UAV). Insome instances, the payload 1204 may be provided on the movable object1200 without requiring the carrier 1202. The movable object 1200 mayinclude propulsion mechanisms 1206, a sensing system 1208, and acommunication system 1210.

The propulsion mechanisms 1206 can include one or more of rotors,propellers, blades, engines, motors, wheels, axles, magnets, or nozzles,as previously described. For example, the propulsion mechanisms 1206 maybe rotor assemblies or other rotary propulsion units, as disclosedelsewhere herein. The movable object may have one or more, two or more,three or more, or four or more propulsion mechanisms. The propulsionmechanisms may all be of the same type. Alternatively, one or morepropulsion mechanisms can be different types of propulsion mechanisms.The propulsion mechanisms 1206 can be mounted on the movable object 1200using any suitable means, such as a support element (e.g., a driveshaft) as described elsewhere herein. The propulsion mechanisms 1206 canbe mounted on any suitable portion of the movable object 1200, such onthe top, bottom, front, back, sides, or suitable combinations thereof.

In some embodiments, the propulsion mechanisms 1206 can enable themovable object 1200 to take off vertically from a surface or landvertically on a surface without requiring any horizontal movement of themovable object 1200 (e.g., without traveling down a runway). Optionally,the propulsion mechanisms 1206 can be operable to permit the movableobject 1200 to hover in the air at a specified position and/ororientation. One or more of the propulsion mechanisms 1200 may becontrolled independently of the other propulsion mechanisms.Alternatively, the propulsion mechanisms 1200 can be configured to becontrolled simultaneously. For example, the movable object 1200 can havemultiple horizontally oriented rotors that can provide lift and/orthrust to the movable object. The multiple horizontally oriented rotorscan be actuated to provide vertical takeoff, vertical landing, andhovering capabilities to the movable object 1200. In some embodiments,one or more of the horizontally oriented rotors may spin in a clockwisedirection, while one or more of the horizontally rotors may spin in acounterclockwise direction. For example, the number of clockwise rotorsmay be equal to the number of counterclockwise rotors. The rotation rateof each of the horizontally oriented rotors can be varied independentlyin order to control the lift and/or thrust produced by each rotor, andthereby adjust the spatial disposition, velocity, and/or acceleration ofthe movable object 1200 (e.g., with respect to up to three degrees oftranslation and up to three degrees of rotation).

The sensing system 1208 can include one or more sensors that may sensethe spatial disposition, velocity, and/or acceleration of the movableobject 1200 (e.g., with respect to up to three degrees of translationand up to three degrees of rotation). The one or more sensors caninclude global positioning system (GPS) sensors, motion sensors,inertial sensors, proximity sensors, or image sensors. The sensing dataprovided by the sensing system 1208 can be used to control the spatialdisposition, velocity, and/or orientation of the movable object 1200(e.g., using a suitable processing unit and/or control module, asdescribed below). Alternatively, the sensing system 1208 can be used toprovide data regarding the environment surrounding the movable object,such as weather conditions, proximity to potential obstacles, locationof geographical features, location of manmade structures, and the like.

The communication system 1210 enables communication with terminal 1212having a communication system 1214 via wireless signals 1216. Thecommunication systems 1210, 1214 may include any number of transmitters,receivers, and/or transceivers suitable for wireless communication. Thecommunication may be one-way communication, such that data can betransmitted in only one direction. For example, one-way communicationmay involve only the movable object 1200 transmitting data to theterminal 1212, or vice-versa. The data may be transmitted from one ormore transmitters of the communication system 1210 to one or morereceivers of the communication system 1212, or vice-versa.Alternatively, the communication may be two-way communication, such thatdata can be transmitted in both directions between the movable object1200 and the terminal 1212. The two-way communication can involvetransmitting data from one or more transmitters of the communicationsystem 1210 to one or more receivers of the communication system 1214,and vice-versa.

In some embodiments, the terminal 1212 can provide control data to oneor more of the movable object 1200, carrier 1202, and payload 1204 andreceive information from one or more of the movable object 1200, carrier1202, and payload 1204 (e.g., position and/or motion information of themovable object, carrier or payload; data sensed by the payload such asimage data captured by a payload camera). In some instances, controldata from the terminal may include instructions for relative positions,movements, actuations, or controls of the movable object, carrier and/orpayload. For example, the control data may result in a modification ofthe location and/or orientation of the movable object (e.g., via controlof the propulsion mechanisms 1206), or a movement of the payload withrespect to the movable object (e.g., via control of the carrier 1202).The control data from the terminal may result in control of the payload,such as control of the operation of a camera or other image capturingdevice (e.g., taking still or moving pictures, zooming in or out,turning on or off, switching imaging modes, change image resolution,changing focus, changing depth of field, changing exposure time,changing viewing angle or field of view). In some instances, thecommunications from the movable object, carrier and/or payload mayinclude information from one or more sensors (e.g., of the sensingsystem 1208 or of the payload 1204). The communications may includesensed information from one or more different types of sensors (e.g.,GPS sensors, motion sensors, inertial sensor, proximity sensors, orimage sensors). Such information may pertain to the position (e.g.,location, orientation), movement, or acceleration of the movable object,carrier and/or payload. Such information from a payload may include datacaptured by the payload or a sensed state of the payload. The controldata provided transmitted by the terminal 1212 can be configured tocontrol a state of one or more of the movable object 1200, carrier 1202,or payload 1204. Alternatively or in combination, the carrier 1202 andpayload 1204 can also each include a communication module configured tocommunicate with terminal 1212, such that the terminal can communicatewith and control each of the movable object 1200, carrier 1202, andpayload 1204 independently.

In some embodiments, the movable object 1200 can be configured tocommunicate with another remote device in addition to the terminal 1212,or instead of the terminal 1212. The terminal 1212 may also beconfigured to communicate with another remote device as well as themovable object 1200. For example, the movable object 1200 and/orterminal 1212 may communicate with another movable object, or a carrieror payload of another movable object. When desired, the remote devicemay be a second terminal or other computing device (e.g., computer,laptop, tablet, smartphone, or other mobile device). The remote devicecan be configured to transmit data to the movable object 1200, receivedata from the movable object 1200, transmit data to the terminal 1212,and/or receive data from the terminal 1212. Optionally, the remotedevice can be connected to the Internet or other telecommunicationsnetwork, such that data received from the movable object 1200 and/orterminal 1212 can be uploaded to a website or server.

FIG. 13 is a schematic illustration by way of block diagram of a system1300 for controlling a movable object, in accordance with embodiments.The system 1300 can be used in combination with any suitable embodimentof the systems, devices, and methods disclosed herein. The system 1300can include a sensing module 1302, processing unit 1304, non-transitorycomputer readable medium 1306, control module 1308, and communicationmodule 1310.

The sensing module 1302 can utilize different types of sensors thatcollect information relating to the movable objects in different ways.Different types of sensors may sense different types of signals orsignals from different sources. For example, the sensors can includeinertial sensors, GPS sensors, proximity sensors (e.g., lidar), orvision/image sensors (e.g., a camera). The sensing module 1302 can beoperatively coupled to a processing unit 1304 having a plurality ofprocessors. In some embodiments, the sensing module can be operativelycoupled to a transmission module 1312 (e.g., a Wi-Fi image transmissionmodule) configured to directly transmit sensing data to a suitableexternal device or system. For example, the transmission module 1312 canbe used to transmit images captured by a camera of the sensing module1302 to a remote terminal.

The processing unit 1304 can have one or more processors, such as aprogrammable processor (e.g., a central processing unit (CPU)). Theprocessing unit 1304 can be operatively coupled to a non-transitorycomputer readable medium 1306. The non-transitory computer readablemedium 1306 can store logic, code, and/or program instructionsexecutable by the processing unit 1304 for performing one or more steps.The non-transitory computer readable medium can include one or morememory units (e.g., removable media or external storage such as an SDcard or random access memory (RAM)). In some embodiments, data from thesensing module 1302 can be directly conveyed to and stored within thememory units of the non-transitory computer readable medium 1306. Thememory units of the non-transitory computer readable medium 1306 canstore logic, code and/or program instructions executable by theprocessing unit 1304 to perform any suitable embodiment of the methodsdescribed herein. For example, the processing unit 1304 can beconfigured to execute instructions causing one or more processors of theprocessing unit 1304 to analyze sensing data produced by the sensingmodule. The memory units can store sensing data from the sensing moduleto be processed by the processing unit 1304. In some embodiments, thememory units of the non-transitory computer readable medium 1306 can beused to store the processing results produced by the processing unit1304.

In some embodiments, the processing unit 1304 can be operatively coupledto a control module 1308 configured to control a state of the movableobject. For example, the control module 1308 can be configured tocontrol the propulsion mechanisms of the movable object to adjust thespatial disposition, velocity, and/or acceleration of the movable objectwith respect to six degrees of freedom. Alternatively or in combination,the control module 1308 can control one or more of a state of a carrier,payload, or sensing module.

The processing unit 1304 can be operatively coupled to a communicationmodule 1310 configured to transmit and/or receive data from one or moreexternal devices (e.g., a terminal, display device, or other remotecontroller). Any suitable means of communication can be used, such aswired communication or wireless communication. For example, thecommunication module 1310 can utilize one or more of local area networks(LAN), wide area networks (WAN), infrared, radio, WiFi, point-to-point(P2P) networks, telecommunication networks, cloud communication, and thelike. Optionally, relay stations, such as towers, satellites, or mobilestations, can be used. Wireless communications can be proximitydependent or proximity independent. In some embodiments, line-of-sightmay or may not be required for communications. The communication module1310 can transmit and/or receive one or more of sensing data from thesensing module 1302, processing results produced by the processing unit1304, predetermined control data, user commands from a terminal orremote controller, and the like.

The components of the system 1300 can be arranged in any suitableconfiguration. For example, one or more of the components of the system1300 can be located on the movable object, carrier, payload, terminal,sensing system, or an additional external device in communication withone or more of the above. Additionally, although FIG. 13 depicts asingle processing unit 1304 and a single non-transitory computerreadable medium 1306, one of skill in the art would appreciate that thisis not intended to be limiting, and that the system 1300 can include aplurality of processing units and/or non-transitory computer readablemedia. In some embodiments, one or more of the plurality of processingunits and/or non-transitory computer readable media can be situated atdifferent locations, such as on the movable object, carrier, payload,terminal, sensing module, additional external device in communicationwith one or more of the above, or suitable combinations thereof, suchthat any suitable aspect of the processing and/or memory functionsperformed by the system 1300 can occur at one or more of theaforementioned locations.

As used herein A and/or B encompasses one or more of A or B, andcombinations thereof such as A and B.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

The invention claimed is:
 1. A method of detecting obstacles using animaging device carried by a movable object, the method comprising:receiving, at one or more processors, first image data of an obstaclefrom the imaging device carried by the movable object, wherein theimaging device is arranged on the movable object so as to have a fieldof view oriented upward or downward relative to the movable object;generating, at the one or more processors, a control signal to controlone or more propulsion units of the movable object to cause the movableobject to move; receiving, at the one or more processors, second imagedata of the obstacle from the imaging device after the movable objecthas moved; calculating, with aid of the one or more processors, adistance between the movable object and the obstacle based on the firstimage data and the second image data; and determining a speed moved bythe movable object between collection of the first image data and thesecond image data, based on the first image data and the second imagedata.
 2. The method of claim 1, further comprising determining a heightof the obstacle above the movable object, based on the first image dataand the second image data.
 3. The method of claim 1, wherein the movableobject is an unmanned aerial vehicle (UAV).
 4. The method of claim 1,wherein the calculating step comprises assessing the first and secondimage data in order to detect a change in a portion of the field of viewof the imaging device occupied by the obstacle in the first image datacollected prior to movement by the movable object and in the secondimage data collected after the movable object has moved.
 5. The methodof claim 1, further comprising determining a horizontal distance or avertical distance moved by the movable object between the collection ofthe first image data and the second image data, based on the first imagedata and the second image data.
 6. The method of claim 1, wherein theimaging device is a monocular camera.
 7. The method of claim 1, whereinthe movable object carries one or more imaging devices that have a fieldof view oriented upward and a field of view oriented downward relativeto the movable object.
 8. The method of claim 1, wherein the obstacle ispositioned at a side of the movable object, and wherein the distancefrom the movable object to the obstacle is based on an angle of view ofthe obstacle within the first image data and the second image data. 9.The method of claim 1, wherein the calculating step comprises: (1)detecting feature points in the first image data and the second imagedata; and (2) determining changes in the feature points between thefirst image data and the second image data.
 10. A system for detectingobstacles using a movable object, the system comprising: an imagingdevice carried by the movable object and configured to obtain image dataof an obstacle, wherein the imaging device is arranged on the movableobject so as to have a field of view oriented upward or downwardrelative to the movable object; one or more propulsion units configuredto effect movement of the movable object; and one or more processorsindividually or collectively configured to: receive first image data ofthe obstacle from the imaging device; generate control signals tocontrol the one or more propulsion units to cause the movable object tomove; receive second image data of the obstacle from the imaging deviceafter the movable object has moved; calculate a distance between themovable object and the obstacle based on the first image data and thesecond image data; and determine a speed moved by the movable objectbetween collection of the first image data and the second image data,based on the first image data and the second image data.
 11. The systemof claim 10, wherein the one or more processors are individually orcollectively configured to determine a height of the obstacle above themovable object, based on the first image data and the second image data.12. The system of claim 10, wherein the movable object is an unmannedaerial vehicle (UAV).
 13. The system of claim 10, wherein the controlsignals cause the movable object to move a horizontal distance or avertical distance.
 14. The system of claim 10, wherein the one or moreprocessors are individually or collectively configured to assess thefirst and second image data in order to detect changes in a portion ofthe field of view of the imaging device occupied by the obstacle priorto and after the movable object has moved.
 15. The system of claim 10,wherein the one or more processors are individually or collectivelyconfigured to determine a horizontal distance or a vertical distancemoved.
 16. A method of detecting obstacles using a movable object, themethod comprising: receiving image data of an obstacle from an imagingdevice carried by the movable object, wherein the imaging device isconfigured to have a field of view oriented upward or downward relativeto the movable object; receiving attitude information for the movableobject; calculating, via one or more processors, a distance between themovable object and the obstacle based on the image data of the obstacleand the attitude information for the movable object; and determining aspeed moved by the movable object between collection of a first imageand a second image from the image data, based on the first image and thesecond image.
 17. The method of claim 16, wherein the calculating stepcomprises determining a portion of the field of view of the imagingdevice that is occupied by the obstacle, based on the image data. 18.The method of claim 16, wherein the movable object is an unmanned aerialvehicle (UAV).
 19. The method of claim 16, further comprising operatingthe movable object within an indoor environment, wherein the image datacomprises images of a ceiling within the indoor environment.
 20. Themethod of claim 16, further comprising operating the movable objectwithin an outdoor environment, wherein the image data comprises imagesof the sky.
 21. The method of claim 16, wherein the field of view of theimaging device is oriented downward relative to the movable object. 22.The method of claim 21, wherein the movable object is operated within anindoor environment and the image data comprises images of a floor withinthe indoor environment.
 23. A system for detecting obstacles using amovable object, the system comprising: an imaging device carried by themovable object and configured to obtain image data of an obstacle,wherein the imaging device is configured to have a field of vieworiented upward or downward relative to the movable object; and one ormore processors individually or collectively configured to: receive theimage data of the obstacle from the imaging device; receive attitudeinformation for the movable object; and calculate a distance between themovable object and the obstacle based on the image data of the obstacleand the attitude information for the movable object; and determine aspeed moved by the movable object between collection of a first imageand a second image from the image data, based on the first image and thesecond image.
 24. The system of claim 23, wherein the one or moreprocessors are individually or collectively configured to determine aportion of the field of view of the imaging device that is occupied bythe obstacle, based on the image data.
 25. The system of claim 23,wherein the movable object is an unmanned aerial vehicle (UAV).
 26. Thesystem of claim 23, wherein the movable object is operated within anindoor environment and the image data comprises images of a ceilingwithin the indoor environment.
 27. The system of claim 23, wherein themovable object is operated within an outdoor environment and the imagedata comprises images of the sky.
 28. The system of claim 23, whereinthe field of view of the imaging device is oriented downward relative tothe movable object.
 29. The system of claim 28, wherein the movableobject is operated within an indoor environment and the image datacomprises images of a floor within the indoor environment.